Blocking Subdomain from Google Crawl and Index
-
Hey everybody, how is it going?
I have a simple question, that i need answered.
I have a main domain, lets call it domain.com. Recently our company will launch a series of promotions for which we will use cname subdomains, i.e try.domain.com, or buy.domain.com. They will serve a commercial objective, nothing more.
What is the best way to block such domains from being indexed in Google, also from counting as a subdomain from the domain.com. Robots.txt, No-follow, etc?
Hope to hear from you,
Best Regards,
-
Hello George, Thank you for fast answer! I read that article and there is some issue with that. if you can see at it, i'd really appreciate it. So the problem is that if i do it directly from Tumblr, it will also block it from Tumblr users. Here is the note right below that option "Allow this blog to appear in search results":
"This applies to searches on Tumblr as well as external search engines, like Google or Yahoo."Also, if i do it from GWT, i'm very concerned to remove URLs with my subdomain because i afraid it will remove all my domain. For example, my domain is abc.com and the Tumblr blog is setup on tumblr.abc.com. So i afraid if i remove tumblr.abc.com from index, it will also remove my abc.com. Please let me know what you think.
Thank you!
-
Hi Marina,
If I understand your question correctly, you just don't want your Tumblr blog to be indexed by Google. In which case these steps will help: http://yourbusiness.azcentral.com/keep-tumblr-off-google-3061.html
Regards,
George
-
Hi guys, I read your conversation. I have similar issue but my situation is slightly different. I'll really appreciate if you can help with this. So i have also a subdomain that i don't want to be indexed by Google. However, that subdomain is not in my control. I mean, i created subdomain on my hosting but it is pointing to my Tumblr blog. So i don't have access to its robot txt. So can anybody advise what can i do in this situation to noindex that subdomain?
Thanks
-
Personally I wouldn't rely just on robots.txt, as one accidental, public link to any of the pages (easier than you may think!) will result in Google indexing that subdomain page (it just won't be followed). This means that the page can get "stuck" in Google's index and to resolve it you would need to remove it using WMT (instructions here). If there were a lot of pages accidentally indexed, you would need to remove the robots.txt restriction so Google can crawl it, and put a noindex/nofollow tags on the page so Google drops it from its index.
To cut a long story short, I would do both Steps 1 and 2 outlined by Federico if you want to sleep easy at night :).
George
-
It would also be smart to add the subdomains in Webmaster Tools in case one does get indexed and you need to remove it.
-
Robots.txt is easiest and quickest way. As a back up you can use the Noindex meta tag on the pages in the subdomain
-
2 ways to do it with different effects:
-
Robots.txt in each subdomain. This will entirely block any search engine to even access those pages, so they won't know what they have inside.
User-Agent:*
Disallow: /
-
noindex tags in those pages. This method allows crawlers to read the page and maybe index (if you set a "follow") the pages to which you link to.or "nofollow" if you don't want the linked pages to be indexed either.
Hope that helps!
-
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Indexing Issues
One of the main pages on my site, http://www.waikoloavacationrentals.com/kolea-rentals/condos, I have been having a hard time getting google to index it correctly or at all. It is one of the top pages on my site and should be in my sub links in google, but it is not even showing up in searches. Any input would be appreciated. The only red flap issue is the number of outgoing links, but that is the way the page is supposed to be. I would assume most real estate listing pages are very similar. Ultimately when you look at traffic, time on page, inbound links, etc. it is one of the top pages on my site in all those categories. Any input would be greatly appreciated.
On-Page Optimization | | RobDalton0 -
Noindex pages being indexed
Hi all Wondering if anyone could offer a pointer on a problem i am having please. I am developing an affiliate store and to prevent problems with duplicate content I have added name="robots" content="NOINDEX,FOLLOW" /> to all the product pages to avoid google penalties. However, Google appears to be indexing product pages. When I do a site: search I see a few hundred product pages in the engine. This is odd as the site has always had noindex on these pages. Even viewing the cache of the indexed page shows the noindex meta tag to be in place. I'm at a loss as to why these pages are being indexed and could do with removing them asap to stop any penalties on the site. Many thanks for any help.
On-Page Optimization | | carl_daedricdigital0 -
Crawl with cach problem
Hello, My Crawl results in Seomoz shows me that i have few thousands of 302 direct problem, bucause it was crawling links like http://www.sposae.com/abito-sposa-g2026-pr-347.html?action=buy_now and it would be redirected automatically to http://www.sposae.com/cookie_usage.php because of cookie not activated from the user. Now I'm wondering if this is an issue to be solved or just a minor thing without impact. Thanks
On-Page Optimization | | angelowei0 -
What´s the penalization extent applied by Google?
Hi! I still don´t get this web site penalization applied by Google due to duplicate content. My site has many of pages that were among the first positions for top keywords (A Photoshop web site). Those pages were linked by sites like LifeHacker, BoingBoing, Microsiervos, SmashingMagazine, John Nack, and many other well known blogs. After mid February 2012 everything went down the drain. I lost half of my traffic and my well ranked pages are now almost nowhere to be found. I have plenty of ads in some pages of my site, and duplicate content (amazon product description only) in other pages of my site. So, the good quality pages my site has, are no longer considered as good quality just because I have some duplicate content or ad filled pages? I´m not complaining. I´m trying to understand this. Google needs to serve good information to their visitors. But since they found some trash in my site, they decide to remove both the trash and the good information from the search engine? That doesn´t sound logical to me. Why don´t they just remove the trash and leave the good content? Of course, I understand that information is added everyday and some may come up with something better than mine, but dropping 40 or more places in the ranking sounds more like a penalty to me. Again, I´m not complaining (although it sounds like I am!), just want to understand the reasons behind this. Thanks, Enrique
On-Page Optimization | | enriquef0 -
Does Google look at page design
Hi everybody, At the moment i'm creating several webshops and websites with the same layout, so visitors can recognize the websites are from the same company. But i was wondering: Does google look at the layout of a webpage that it's not a copy of another website? This because loads of website have the same wordpress/joomla templates etc, or doesn't this effect rankingpositions? Thank you,
On-Page Optimization | | iwebdevnl0 -
Google Instant Preview
Is there a way of having videos show up in google instant preview? Right now all I am getting is a blank space
On-Page Optimization | | casper4340 -
Google Place Pages - Definitive Guide?
Hi good folk of SEOmoz, I've recently updated a number of place pages for my client who is an estate agent. I have to admit that they were a little keyword stuffed, but at the time my competitor; who ranks first, was filled with keywords. For example in their title it states "[company name] estate agent [location]" I thought that this was a big no-no and it should not be done? They also have keywords stuffed into categories and description. Im confused on how they are getting away with this? I have searched on the Google place page support forum and can not find a definitive guide on the exact rules. Could anybody be so kind to help me out? i.e Where to put keywords and locations in the title and categories Thankyou.
On-Page Optimization | | Lakeside0 -
Google indexing Internal Search Results
Greeting, Currently I have noticed that Google is starting to index our internal search page results. Should I block those pages in our robot txt file or have you ever heard of any websites that actually gained traffic or rank by letting Google index those pages? Thanks
On-Page Optimization | | Tonyd230