Robots.txt disallow subdomain
-
Hi all,
I have a development subdomain, which gets copied to the live domain. Because I don't want this dev domain to get crawled, I'd like to implement a robots.txt for this domain only. The problem is that I don't want this robots.txt to disallow the live domain. Is there a way to create a robots.txt for this development subdomain only?
Thanks in advance!
-
I would suggest you talk to the developers as Theo suggests to exclude visitors from your test site.
-
The copying is a manual process and I don't want any risks for the live environment. A Httphandler for robots.txt could be a solution and I'm going to discuss this with one of our developers. Other suggestions are still welcome of course!
-
Do you ftp copy one domain to the other? If this is a manual process the excluding the robots.txt that is on the test domain would be as simple as excluding it.
If you automate the copy and want code to function based on base url address then you could create a Httphandler for robots.txt that delivered a different version based on the request url host in the http request header.
-
You could use enviromental variables (for example in your env.ini or config.ini file) that are set to DEVELOPMENT, STAGING, or LIVE based on the appropriate environments the code finds itself in.
With the exact same code, your website would either be limiting IP addresses (on the development environment) or allow all IP addresses (in the live environment). With this setup you can also set different variables per environment such as the level of detail that is shown in your error reporting, connect to a testing database rather than a live one, etc.
[this was supposed to be a reply, but I accidentely clicked the wrong button. Hitting 'Delete reply' results in an error.]
-
Thanks for your quick reply, Theo. Unfortunately, this htpasswd will also get copied to the live environment, so our websites will get password protected live. Could there be any other solution for this?
-
I'm sure there is, but I'm guessing you don't want any human visitors to go to your development subdomain and view what is being done there as well? I'd suggest you either limit the visitors that have access by IP address (thereby effectively blocking out Google in one move) and/or implement a .htpasswd solution where developers can log in with their credentials to your development area (which blocks out Google as well).
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Choosing root domain for a subdomain
We own two root domains in the .edu space. According to Open Site Explorer, one has a domain authority of 76, while the other has a DA of 94. We operate a collection of degree microsites as subdomains of the lower-ranking root domain, e.g. www.degreename.domain76.edu. All other things being equal, would these sites benefit if we migrated them to www.degreename.domain94.edu? The question seems to hinge on whether subdomains inherit any of the root domain's authority, and the answers I have seen to that question are "sometimes" and "maybe". Lastly, as an alternative, would we realize greater SEO improvements by moving the degrees to a directory structure under domain94, i.e. www.domain94.edu/degrees/degree-name? Thank you for your help!
Intermediate & Advanced SEO | | UWPCE0 -
Should I disallow all URL query strings/parameters in Robots.txt?
Webmaster Tools correctly identifies the query strings/parameters used in my URLs, but still reports duplicate title tags and meta descriptions for the original URL and the versions with parameters. For example, Webmaster Tools would report duplicates for the following URLs, despite it correctly identifying the "cat_id" and "kw" parameters: /Mulligan-Practitioner-CD-ROM
Intermediate & Advanced SEO | | jmorehouse
/Mulligan-Practitioner-CD-ROM?cat_id=87
/Mulligan-Practitioner-CD-ROM?kw=CROM Additionally, theses pages have self-referential canonical tags, so I would think I'd be covered, but I recently read that another Mozzer saw a great improvement after disallowing all query/parameter URLs, despite Webmaster Tools not reporting any errors. As I see it, I have two options: Manually tell Google that these parameters have no effect on page content via the URL Parameters section in Webmaster Tools (in case Google is unable to automatically detect this, and I am being penalized as a result). Add "Disallow: *?" to hide all query/parameter URLs from Google. My concern here is that most backlinks include the parameters, and in some cases these parameter URLs outrank the original. Any thoughts?0 -
Subdomains vs. Subfolders vs. New Site
Hello geniuses!!! Here's my Friday puzzle: We have a plastic surgery client who already has a website that's performing fairly well and is driving in leads. She is going to be offering a highly specialized skincare program for cancer patients, and wants a new logo, new website and new promo materials all for this new skincare program. So here's the thing - my gut reaction says NO NEW WEBSITE! NO SUBDOMAIN! because of everything I've read about moving things on and off subdomains, etc (I just studied this: http://moz.com/blog/subdomains-vs-subfolders-rel-canonical-vs-301-how-to-structure-links-optimally-for-seo-whiteboard-friday). And, why wouldn't we want to use the authority of her current site, right? While she doesn't necessarily have a high authority domain - we're not talking WebMD, here - she does have some authority that we've built over time. But, because this is a pretty separate product from her general plastic surgery practice, what would you guys do? Since we'll be creating a logo and skincare "look and feel" for this product, and there will likely be a lot of information involved with it, I don't think we'll be able to just create one page. Is it smart to: a) build a separate site in a subfolder of her current site? (plasticsurgerypractice.com/skincare) b) build a subdomain? (skincare.plasticsurgerypractice.com) c) build her a new site (plasticsurgeryskincare.com)
Intermediate & Advanced SEO | | RachelEm0 -
Using subdomains for related landing pages?
Seeking subdomain usage and related SEO advice... I'd like to use multiple subdomains for multiple landing pages all with content related to the main root domain. Why?...Cost: so I only have to register one domain. One root domain for better 'branding'. Multiple subdomains that each focus on one specific reason & set of specific keywords people would search a solution to their reason to hire us (or our competition).
Intermediate & Advanced SEO | | nodiffrei0 -
Fourth and Third Level Subdomain Interlinking
Hi everyone. I have a hopefully interesting client question I wanted to pose. I do work for a company with three distinct locations that have unique offerings, service areas, etc. I wouldn't want to see the three locations as subfolders (example company.com/locationone/, company.com/locationtwo/) of one site. They are large and unique presences. Fortunately, they did not organize their locations in such a way, and currently have their locations organized as subdomains, as in locationone.company.com, locationtwo.company.com. I might have preferred locationone.com, locationtwo.com, etc., but that is what I am working with. Their developer has been building new content on fourth level domains, as in newcontent.locationone.company.com and newcontent2.locationone.company.com. In one case one of these fourth level domains also contains a different but parallel checkout process to the one already present on third level domain locationone.company.com. I am looking for advice on how to interlink these sites, and whether to discourage them from building out new fourth level domains (newcontent3.locationone.company.com, etc.) or even to get rid of the current fourth level domains altogether. I'm not sure if the fact that they're subdomains and not subfolders matters as much as it used to. Is this a case of subdomain phobia, or are my concerns justified? Any special advice on dealing with interlinking across fourth, third, and second level domains? Thanks!
Intermediate & Advanced SEO | | rlevers0 -
1200 pages no followed and blocked by robots on my site. Is that normal?
Hi, I've got a bunch of notices saying almost 1200 pages are no-followed and blocked by robots. They appear to be comments and other random pages. Not the actual domain and static content pages. Still seems a little odd. The site is www.jobshadow.com. Any idea why I'd have all these notices? Thanks!
Intermediate & Advanced SEO | | astahl110 -
Robots.txt unblock
I'm currently having trouble with what appears to be a cached version of robots.txt. I'm being told via errors in my Google sitemap account that I'm denying Googlebot access to the entire site. I uploaded clean and "Allow" robots.txt yesterday, but receive the same error. I've tried "Fetch as Googlebot" on the index and other pages, but still the error. Here is the latest: | Denied by robots.txt |
Intermediate & Advanced SEO | | Elchanan
| 11/9/11 10:56 AM | As I said, there in not blocking on the robots.txt for 24 hours. HELP!0 -
Why specify robots instead of googlebot for a Panda affected site?
Daniweb is the poster child for sites that have recovered from Panda. I know one strategy she mentioned was de-indexing all of her tagged content, fo rexample: http://www.daniweb.com/tags/database Why do you think more Panda affected sites specifying 'googlebot' rather than 'robots' to capture traffic from Bing & Yahoo?
Intermediate & Advanced SEO | | nicole.healthline0