Disallow indexing of ALL subdomains
-
I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:
project1.domain.com
project2.domain.com
project3.domain.com
...Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.
Is this possible?
-
Hello there!
Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).
Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.
I hope this help!
-
Nobody?
-
Thanks for the link, but I don't think that will solve my problem?
When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?
-
You should edit your .htaccess file as is described here:
http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-repositI hope i answered your question
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do sites built with WordPress work in China get indexed by Baidu?
I think that WordPress.com is banned, but are websites that are created by WordPress get banned?
On-Page Optimization | | CoGri0 -
Using a subdomain to improve rankings
I have a pretty simple landing page that at present targets several keywords. Is a good strategy to add a blog to the site on a subdomain with pages targeting individual keywords? Will this help the main domain rank? Also is the best strategy to focus on building links to the home page or individual pages on the subdomain? As long as each page on the subdomain links back to the home page it should pass link juice back to the home page shouldn't it? Thanks in advance!
On-Page Optimization | | SamCUK0 -
Subdomain vs. Root Domain design
I know that there are penalties if you have a navigation item in your categories that links to another site (root domain) that yields the same design as the original link. Is there any negative SEO implications with having a subdomain with the same design as the root domain i.e. root.com vs. store.root.com And the navigation functionality within will have links to both store.root.com as well as root.com directories? Thanks in advance.
On-Page Optimization | | customerparadigm.com0 -
Issues with Product Pages Getting Index In Google
I just started working here the other week and one of the big issue is that a lot of the product pages are not getting index in google. We have an xml.gz site map they submitted a long time ago. My guess is it might be something with not enough content on the pages? Here are a few example of pages that are not getting index in google. http://www.rockymountainatvmc.com/p/43/-/439/716/-/33097/Alpinestars-Dual-Motorcycle-Gloves http://www.rockymountainatvmc.com/p/47/-/201/803/-/28948/Camelbak-Blowfish-2013 http://www.rockymountainatvmc.com/p/46/-/203/836/-/6996/MSR-Head-Case http://www.rockymountainatvmc.com/p/44/54/208/764/80/1220/Galfer-Brake-Pad-Sintered-Metal There are 100's that are not indexed just trying to figure out what we need to do! We are working on new content to them all but we have over 5000 products so it will take a long time. We also have the reviews on the pages and are looking at starting a Q&A on page to help get more unique content.
On-Page Optimization | | DoRM0 -
Modifying Well Established & Well Indexed Content
I have a page that is very well indexed and has a 1st position ranking in google. It is the best landing page in my site. That being said, it's several years old and I honestly think it could be better. The images could be enlarged, the the images could have fancy box enlargements instead of just linking out to flickr, there could be more content about follow up projects that people have done. I'm noteably nervous about changing such a clutch piece of content on my site. I do want to improve the content for users, not just make it more SEO friendly (it's already SEO'd), but I'm afraid that any change could cause a set back in ranking. Am I being afraid of nothing, should I just go for it and improve my content, or should I be extra cautious when editing well indexed content like this? Thanks for the advice
On-Page Optimization | | CPollock0 -
Is it ok to point internal links to index.html home page rather than full www
I thought I saw this somewhere on SEOmoz before but I was so busy by the time I got around to work on my SEO on my site, I realized I have this happening and can't recall if it is a problem which takes away from my ranking. If my www.website.com is ranking well but I have internal menu links pointing to www.website.com/index.html instead of www.website.com will that take away from my www.website.com rankings? Should I change all my menu links that point to /index.html to the full website url path www.website.com ?
On-Page Optimization | | Twinbytes0 -
0 urls indexed in GWT, many found with site: command
Hi, This is happening with a brand new site. We have created sitemaps and submitted them to Google Webmaster Tools. GWT says sitemaps are ok, "x" number of urls submitted, but no urls indexed. When I check in Google with site:domain.com I see that many of the urls are already indexed. Why this discrepancy between GWT and reality? Thanks for your time!
On-Page Optimization | | gerardoH0 -
ON SITE SEARCH INDEXED BY GOOGLE - no follow or no index
Google indexes alll our internetal searches: search box is brand - clothes types - size type - and for each page it creates a page that which creates duplicate page title and unnecessary content. Should I do a nofollow on the advance search or a no index. Many thanks for the info. Sonja
On-Page Optimization | | reallyitsme0