Robots.txt on subdomains
-
Hi guys!
I keep reading conflicting information on this and it's left me a little unsure. Am I right in thinking that a website with a subdomain of shop.sitetitle.com will share the same robots.txt file as the root domain?
-
That's about as comprehensive an answer as I could have hoped for. Thanks Ryan, really appreciated.
-
Mostly no. I say 'mostly' because a lot of times when you look at a site using www and no-www if both of those work they're almost always pulling files from the same location (hence the warnings around duplicate content), so both www.domain.com/robots.txt and domain.com/robots.txt are going to work. This is the dominant example of a subdomain sharing a robots.txt file. However, on domains that are set up as their own subdomains they have different robots.txt. Take a look at the many differences between subdomain1-1000.wordpress.com/robots.txt vs wordpress.com/robots.txt. If you set up a subdomain that isn't just a reflection of your root domain, then you'll need to create a robots.txt file as well. Cheers!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Product Subdomain Outranking "Marketing" Domains
Hello, Moz community! I have been puzzling about what to do for a client. Here is the challenge. The client's "product"/welcome page lives at www.client.com this page allows the visitor to select the country/informational site they want OR to login to their subdomain/install of the product. Google is choosing this www.client.com url as the main result for client brand searches. In a perfect world, for searchers in the US, we would get served the client's US version of the information/marketing site which lives at https://client.com/us, and so on for other country level content (also living in a directory for that country) It's a brand new client, we've done geo-targeting within the search console, and I'm kind of scared to rock the boat by de-indexing this www.client.com welcome screen. Any thoughts, ideas, potential solutions are so appreciated. THANKS! Thank you!
Technical SEO | | SimpleSearch0 -
Migrating to new subdomain with new site and new content.
Our marketing department has decided that a new site with new content is needed to launch new products and support our existing ones. We cannot use the same subdomain(www = old subdomain and ww1 = new subdomain)as there is a technically clash between the windows server currently used, and the lamp stack required to run the new wordpress based CMS and site. We also have an aging piece of SAAS software on the www domain which is makes moving it to it's own subdomain far too risky. 301's have been floated as a way of managing the transition. I'm not too keen on that idea due to the double effect of new subdomain and content, and the SEO impact it might have. I've suggested uploading the new site to the new subdomain while leaving the old site in place. Then gradually migrating sections over before turning parts of the old site off and using a 301 at that point to finalise the move. The old site would inform user's there is a new version and it would then convert them to the new site(along with a cookie to auto redirect them in future.) while still leaving the old content in place for existing search traffic, bookmarks and visitors via static URLs. Before turning off sections on the old site we would create rel canonicals to redirect to the new pages based on a a mapped set of URLs(this in itself concerns me as the rel canonical is essentially linking to different content). Would be grateful for any advice on whether this strategy is flawed or whether another strategy might be more suitable?
Technical SEO | | Rezza0 -
Google ignores Meta name="Robots"
Ciao from 24 degrees C wetherby UK, On this page http://www.perspex.co.uk/products/palopaque-cladding/ this line was added to block indexing: But it has not worked, when you google "Palopaque PVC Wall Cladding" the page appears in the SERPS. I'm going to upload a robots txt file in a second attempt to block indexing but my question is please:
Technical SEO | | Nightwing
Why is it being indexed? Grazie,
David0 -
Is a shorter subdomain better?
For example, consider the two subdomains below: learn.bluelinkerp.com learnmore.bluelinkerp.com Would there likely be an appreciable difference between the two based on length alone? Also, would it be better to use a term that relates to our product in general such as "software.bluelinkerp.com"? Thanks!
Technical SEO | | BlueLinkERP0 -
Multiple Subdomains, my worst seo mistake. now what should i do?
Hello Everyone, I have been running www.designzzz.com from lats 3 years now. and was doing extremely good with a PR 6 and 800K+ traffic monthly, but 6 months ago it started falling and falling badly.. now i am down to 350K total impressions :{ per month. I have been blaming penguin for this and been talking to google reps continously over it. they assured me that my site is not under any type of manual spam etc. Then i begin think and i realized taht was exactly the time when i launched a few subdomains as sub parts of my site like coding.designzzz.com , wordpress.designzzz.com , photograph.designzzz.com , shop.designzzz.com in the making... now is the part that i can't undo these subdomains.. what should i do ? my search traffic is almost killed. I seriously need insight on this guys : thanks in advance! Ayaz
Technical SEO | | wickedsunny10 -
Rankings of Subdomains vs. Main Domain
Here's a puzzler... Our main domain (www.ides.com) doesn't appear in Google (but does on Bing and other engines). We think this is due to duplicate content which we're fixing. However our website's subdomains continue to appear prominently in SERPs, even on Google - here are some examples: IDES Prospector = prospector.ides.com IDES = support.ides.com Cycolac FR15 = catalog.ides.com Why would Google penalize a main domain and its subdomains?
Technical SEO | | Prospector-Plastics0 -
Restricted by robots.txt does this cause problems?
I have restricted around 1,500 links which are links to retailers website and links that affiliate links accorsing to webmaster tools Is this the right approach as I thought it would affect the link juice? or should I take the no follow out of the restricted by robots.txt file
Technical SEO | | ocelot0 -
Subdomain Robots.txt
If I have a subdomain (a blog) that is having tags and categories indexed when they should not be, because they are creating duplicate content. Can I block them using a robots.txt file? Can I/do I need to have a separate robots file for my subdomain? If so, how would I format it? Do I need to specify that it is a subdomain robots file, or will the search engines automatically pick this up? Thanks!
Technical SEO | | JohnECF0