Does Google respect User-agent rules in robots.txt?
-
We want to use an inline linking tool (LinkSmart) to cross link between a few key content types on our online news site.
LinkSmart uses a bot to establish the linking.
The issue: There are millions of pages on our site that we don't want LinkSmart to spider and process for cross linking.
LinkSmart suggested setting a noindex tag on the pages we don't want them to process, and that we target the rule to their specific user agent.
I have concerns. We don't want to inadvertently block search engine access to those millions of pages. I've seen googlebot ignore nofollow rules set at the page level. Does it ever arbitrarily obey rules that it's been directed to ignore?
Can you quantify the level of risk in setting user-agent-specific nofollow tags on pages we want search engines to crawl, but that we want LinkSmart to ignore?
-
Does Google respect User-agent rules in robots.txt?
Yes
I've seen googlebot ignore nofollow rules set at the page level.
Google honors the nofollow rules set at the page level. The issue is there may be other links on your site or elsewhere on the web that Google will find and follow those links.
Robots.txt is the absolute last means to use for blocking pages. You should not block a page with robots.txt unless you have exhausted all other options. A more appropriate method of keeping a page out of the index is the noindex tag. If you use the tag appropriately, Google will honor the tag.
-
Hi,
I would advise to block the directories which the files sit in in robots.txt, over adding no index tags to specific pages.
Yet then this would also leave these pages to not be indexed by Google, other search engines and also this Link Smart software you are referring to.
The thing is if you add a no index tag or if you add a robots .txt block to pages it will also block all search engines too.
So yes their is some risk involved, you have to do things carefully around this area.
Kind Regards,
James.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
HTTPS, free certificates and Google seo
As we all know, Google decided to give some weight in rankings to encrypted sites. SSL certificate providers are quite a lot out there, but there is also a free alternative (Let’s Encrypt). I've been going over some of these services and compared Godaddy's and the above certificates, and there isn't much of a difference. The question is: using a free certificate would have the same effect as a paid one? I would rather pay for one than be punished for using a free one, but free would be good too.
On-Page Optimization | | anitawapa
What's your take on this? Thank you0 -
Is it bad to include google Maps in footer?
We have 5 locations and we were thinking about including a map for each location in the footer. These would be set-up as no-follow links. They could potentially enhance user experience but it also increases size of footer. Right now there are just basic links to pages (sitemap, terms, etc), contact info, social links, and contact form. If we did the maps it would also include link to the individual location pages. Not sure if we are doing too much in footer or need to just keep it basic. Thanks for the help!
On-Page Optimization | | Restore0 -
Google Xml Sitemaps
Which plugin is good to use to create and submit my sitemap: sitemap from yoast or google xml sitemap plugin?
On-Page Optimization | | Sebastyan22
Which one is better? I already saw this video but I get an error when I submited it to webmaster tools and I don't know why:http://www.quicksprout.com/university/how-to-set-up-and-optimize-a-sitemap/_''Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead.''_Thank you !0 -
How to optimize Google Places listing?
I have a couple friends who need more traffic from their Google Places listing. They are in very niche businesses. One is a home remodeling contractor and one is a music teacher. Does anyone know some good services to help optimize their listings? I feel I've helped them out as much as I can by getting real reviews to their Google Places page. But I feel that someone who's had the experience in Google Places optimization can pick up on something I'm overlooking.
On-Page Optimization | | RicktheMarkt0 -
Getting Google to see major changes?
I am making some big changes to my site based on Moz's advice. How can I get google to see all of them? I submitted them to Google Webmasters. Is that the best thing to do? How long until Google updates all my pages?
On-Page Optimization | | dealblogger0 -
Will Google re-index the page if content is changed/improved?
Hello,
On-Page Optimization | | wickedsunny1
i have a little question, it seems most of my posts which are 3 years old or even more, all have almost same type of 2-3 lines of text at post starting and then images roundups etc.
which recently stopped getting any traffic from google. Probably becuase of lack in text etc.
So if i now edit those old pages, will they be reindexed in google with this new content/data?
thanks in advance.
cheers0 -
Will Google handle "this not that" pages differently?
If you create pages about "try keyword1 not keyword2" will there be any barriers to getting the pages ranked for keyword2? Example: You have furnished rental units in a small town, and you offer nightly/weekly rentals. You want to rank for "town hotel" since you offer the same service as a hotel. Since you're not really a hotel, you create a page called "Better than a hotel: Town nightly rental units". Anyone know if Google has an algorithm to detect this (they would have to detect the meaning of the words you were using and know that you were promoting something other than a hotel) and determine you're not really relevant to "town hotel" and not rank you well? I think they probably do not, as I've seen things like Google Adsense Alternatives articles ranking well for the term Google Adsense, or Boycott Godaddy sites ranking well for the term godaddy. But I would like to hear any evidence or facts others know of.
On-Page Optimization | | AdamThompson0 -
Is keyword stuffing relative to total copy length or an absolute rule?
My seomoz report has warned me about 38 occurrences of a keyword on a page, is that always too many? 19 of them are in my reviews tab, "I love KEYWORD", etc. I expect to get more reviews and more keyword mentions. If I had to keep that page to 15 or fewer, I would have no reviews tab.
On-Page Optimization | | Brocberry0