Blocking Test Pages Enmasse on Sub-domain
-
Hello,
We have thousands of test pages on a sub-domain of our site. Unfortunately at some point, these pages were visible to search engines and got indexed. Subsequently, we made a change to the robots.txt file for the test sub-domain. Gradually, over a period of a few weeks, the impressions and clicks as reported by Google Webmaster Tools fell off for the test. sub-domain.
We are not able to implement the no index tag in the head section of the pages given the limitations of our CMS.
Would blocking off Google bot via the firewall enmasse for all the test pages have any negative consequences for the main domain that houses the real live content for our sites (which we would like to of course remain in the Google index).
Many thanks
-
If you want nothing on that test subdomain indexed, verify that subdomain as its own site in Google Webmaster Tools, exclude that subdomain from being indexed in robots.txt, then request removal of that site (subdomain) in GWT.
And consider setting up a page monitor like https://polepositionweb.com/roi/codemonitor/index.php on the robots.txt of your test site (and live site). It'll check the contents of those pages once a day, and email you if there's a change. Handy if there are multiple people working on the site.
-
I'm a bit confused. Didn't blocking the test subdomain with the robots.txt already accomplish what you are trying to do? Or are the test pages still somehow indexed? Or is your main site affected by the robots.txt? Anyway, I would suggest using the .htaccess file to block search engines from accessing the subdomain rather than a firewall - http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-reposit
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving Shopify from a Sub Domain to the Full Domain
Apologies if this has been asked before. Currently we have a Shopify shop on a subdomain shop.lucybee.com a blog on subdomain blog.lucybee.com and the full domain of course. We'd now like to move everything onto Shopify on the full domain. Therefore the blog rolls into Shopify. We'll manually move pages into Shopify. Any advice or links to resources on how to manage would be gratefully received. Thank you , Jim
Technical SEO | | LucyBee0 -
Google page speed testing failures
When running our domain through google's page speed insights I'm getting the error message 'An error occurred while fetching or analyzing the page' at around 65% load. I'm concerned it's affecting our organic rankings. The domain is https://www.scottscastles.com/ When testing in https://testmysite.withgoogle.com/ it is also failing at around 70% with the message 'It's taking longer than expected. You can leave this tab open and check back in a little while. We'll have your results soon.' but the results never come. I've tried testing on a few different speed testing sites without failures (https://tools.pingdom.com, https://gtmetrix.com, https://www.webpagetest.org and a few others). We’re stumped as everything appears correct and was working but now isn't. Is this Google or us, or a combination of the two? Any help greatly appreciated!
Technical SEO | | imaterus0 -
What should I consider before setting up a sub domain?
Morning all! We've just been approached by IT. They've been asked to develop an online 'portal' where clients can upload and download materials. IT will be developing a portal that sits on the company network perimeter (hosted on our internal servers). The concept is that 3<sup>rd</sup> parties can get and update information in regards to progressing cases, the first use will be for agencies who will retrieve records via the portal and then post reports after a consultation. however I would like to have an automatic link to forward to the portal from the web address: oursite.com/dave We will look to create robot.txt and anything else to prevent from listings/indexes. Does any of the above mess with your SEO? The Directors have asked if they can have this on a sub-domain of our site. Is this wise? And, are there any major SEO considerations for my team to worry about? Better still, have any of you had to deal with this before? If so, what happened? All the best, John
Technical SEO | | Muhammad-Isap0 -
<sub>& <sup>tags, any SEO issues?</sup></sub>
Hi - the content on our corporate website is pretty technical, and we include chemical element codes in the text that users would search on (like S02, C02, etc.) A lot of times our engineers request that we list the codes correctly, with a <sub>on the last number. Question - does adding this code into the keyword affect SEO? The code would look like SO<sub>2</sub>.</sub> Thanks.
Technical SEO | | Jenny10 -
Does a sub-domain benefit a domain...
I have seen some mixed comment as the whether a sub-domain would benefit from the authority built up by its domain... but does a domain benefit from a sub-domain?
Technical SEO | | Switch_Digital0 -
Followed Linking Root Domains and No Followed Linking Domains
If you have more NoFollowed Linking Root Domains than Followed Linking Root Domains is that a problem?
Technical SEO | | INN0 -
New domain
Hi, I have a domain with no keywords on it, and I´ve been using it for years. Now I bought another domain with the keyword on it. I whant to work on seo for the second domain, with the keyword. What is the better way to work this out? 301? Duplicate de site? redirect in another way?
Technical SEO | | mgfarte0 -
I have 15,000 pages. How do I have the Google bot crawl all the pages?
I have 15,000 pages. How do I have the Google bot crawl all the pages? My site is 7 years old. But there are only about 3,500 pages being crawled.
Technical SEO | | Ishimoto0