Disallow indexing of ALL subdomains
-
I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:
project1.domain.com
project2.domain.com
project3.domain.com
...Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.
Is this possible?
-
Hello there!
Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).
Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.
I hope this help!
-
Nobody?
-
Thanks for the link, but I don't think that will solve my problem?
When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?
-
You should edit your .htaccess file as is described here:
http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-repositI hope i answered your question
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I check spam score and non indexed links
Hi I am working on a project in which I have a local services website <a target="_self">www.thepcrrg.com</a> I want to be sure its links are getting indexed or not. Please help me.
On-Page Optimization | | Jioiebytt2 -
No index for http version of website
Hi, I've had a message from Google search console to say the sitemap for the http version of my site is tagged as no index. As the https version is indexed, do I need to change the http version to be indexed as well? Do I need to keep the http version of the site in search console alongside the https version, or should I remove it? Advice appreciated!
On-Page Optimization | | Robingoodlad0 -
Subdir vs Subdomain vs ccTLD
Hello Community, I'm running a .com website with India targetted audience on Webmastertools from 2 years. Now I want to start a separate zone for US. .us domain is not available for my domain right now. So what is your suggestion for best results: 1. domain/us (Subdirectory) 2. us.domain (Subdomain) 3. new domain (separate website) I think Subdomain can work great if I will pass place a header link sitewide. Or that can reduce authority as mentioned on few blogs? Let me know your thoughts
On-Page Optimization | | sah900 -
Index or No Index (Panda Issue)
Hi, I believe our website has been penalized by the panda update. We have over 9000 pages and we are currently indexing around 4,000 of those pages. I believe that more than half of the pages indexes have either thin content. Should we stop indexing those pages until we have quality page content? That will leave us with very few pages being indexed by Google (Roughly 1,000 of our 9,000 pages have quality content). I am worried that we would hurt our organic traffic more by not indexing the pages than by indexing the pages for google to read. Any help would be greatly appreciated. Thanks, Jim Rodriguez
On-Page Optimization | | dustyabe0 -
Should I block google indexing "search.php"
My question is I have a search page on our website , you can search by date, number of people staying and so on, I am just wondering should block this in the robots.txt ? Because we have pretty URL'S already for searching by county and searching by towns. I cannot see any benefit of having e.g "search/search.php?sp_dateFrom=16%2F12%2F2015&sp_dateTo=23%2F12%2F2015&sec_drop%5B%5D=727&spesh_town_id=764&q=&occupants=5&bedrooms=3&submit=SEARCH#search" indexed. Would I be correct in doing this ?
On-Page Optimization | | McCaldin0 -
Are a lot auf tag-sites in the index a bad signal for low quality? (Panda Update)
Hello everybody, first of all please excuse my bad english. I'm from Germany - I try my best. 😉 The case: I have a Wordpress SEO project which rankings very well. A this moment I have all "archive sites" like "archive", "category" und "tags" indexed. I use the more-Tag for every archive/category/tag site - so duplicate content ist not really a problem, but in view of the Panda Update, which surely arrives in Germany soon, I wonder if all this Tag/Archive/Category Sites in the index maybe seen as low quality und can hurt the ranking of my whole site. Low quality because: With using the more-tag the site are just a list of internal links with content snippets. I have 500 articles und 700 Tag Site (all in the index). So my fear is when google (with Panda Update) looks at my site und sees all this (maybe) low quality tag-sites in the index I get penalised because there is not a good proportion between my normal (good quality) Articles und the archive/tag sites. I hope you guys can understand my thoughts. Do I have a legitimate fear that the mass of tag-site in the index could be problem? Are there any data from the USA, how blogs mit Tag-Site in the Index rank after the Panda Update or if sites which contains of internal Links mit content snippets - like these tag site - are low quality in Google eyes? Or I'm worring to much? Thank you very much! Oliver
On-Page Optimization | | channelplus0 -
Page Indexing
Hello All Nice easy question! I've made some on page changes to page titles, content, H1s etc but wanted to know if there was a way to check if Google has reindexed the page since the changes were made? I appreciate the different factors that will help improve your crawl rate like new content, external links, domain authority etc. I made these changes around 2 weeks ago. Google has cached the pages since I made the changes but not picked up on the new page titles in the search results. Cheers Todd
On-Page Optimization | | todd75850 -
Does Frequency of content updates affect likelyhood outbound links will be indexed?
I have several pages on our website with low pr, that also themselves link to lots and lots of pages that are service/product specific. Since there are so many outbound links, I know that the small amount of PR will be spread thin as it is. My question is, if I were to supply fresh content to the top level pages, and change it often, would that influence whether or not google indexes the underlying pages? Also if I supply fresh content to the underlying pages, once google crawls them, would that guarantee that google considers them 'important' enough to be indexed" I guess my real question is, can freshness of content and frequency of update convince google that the underlying pages are 'worthy of being indexed', and can producing fresh content on those pages 'keep google's interest', so to speak, despite having little if any pagerank.
On-Page Optimization | | ilyaelbert0