Why are "noindex" pages access denied errors in GWT and should I worry about it?
-
GWT calls pages that have "noindex, follow" tags "access denied errors."
How is it an "error" to say, "hey, don't include these in your index, but go ahead and crawl them."
These pages are thin content/duplicate content/overly templated pages I inherited and the noindex, follow tags are an effort to not crap up Google's view of this site.
The reason I ask is that GWT's detection of a rash of these access restricted errors coincides with a drop in organic traffic. Of course, coincidence is not necessarily cause.
Should I worry about it and do something or not?
Thanks... Darcy
-
I am a little surprised, because having those pages as "noindex, follow" should not bring GWT to flag them as errors.
Monica is correct in addressing google flag anything than 200 as errors, but... Your page with "noindex, follow" should return a HTTP code of 200. If it is returning anything else, it's probably wrong, and you should analyze why is doing it.
My religion has a law saying that GWT should return no errors, point. I have also witnessed few times a correlation between lowering GWT errors count to 0 and an improve in SERP ranking; but I have no proof one is causing the other.
-
I had a similar issue where my sitemap and my robots.txt didn't match properly and they were causing a slew of errors to show up. Everything falls under a crawler error but "should" clean itself up as its being indexed. I resubmitted an updated sitemap that matched my robots.txt and I have gotten rid of the errors.
Google also states that these errors don't directly hurt your ranking, but they can indirectly hurt because of user experience. You can always double check and see if the pages are being indexed by doing a "site:" search in google and checking if those pages exist.
Now, the errors are somewhat of a blessing. We had a design firm who redid our website and they had contracted an SEO "expert" to optimize the site before launch. They launched our website, and the next day I open up GWMT and our entire website was still under "noindex". The forgot to take the noindex from the dev site off of our main site.
Also I would consider just redirecting the thing content all together.
EDIT: And again Ryan sneaks in before me!!!!!!!!
-
Thumbs up to Monica's answer. I'd just add that you could redirect some of those pages to thin out the use of no index if possible, but it sounds like you've kept them around as they're marginally useful. You can also click the 'ignore' button for given error messages and they'll go away.
-
No. I wouldn't worry about it. Google calls them errors, the same as a 404 error. To them an error is anything that returns a code other than 200. I have hundreds of noindex pages on my site and it doesn't hurt. I believe it helps because it removes duplicate content and eliminates bad user experiences.
I have always thought that it is Google's way of double checking to make sure that the Webmaster is aware those pages are blocked. There have been times that I found URLs in there that weren't supposed to be, and contrarily found missing URLs as well. Its checks and balances in my opinion.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should you 'noindex' Checkout Pages?
Today I was reviewing my Moz analytics and suddenly noticed 1,000 issues with pages without a meta description. I reviewed the list and learned it is 1,000 checkout pages. That's because my website has thousands of agency pages from which you can buy a product, and it reflects that difference on each version of the checkout. So, I was thinking about no-indexing (but continuing to 'follow') these checkout pages, but wondering if it has any knock-on effects I may be unaware of? Any assistance is much appreciated. Luke
Intermediate & Advanced SEO | | Luke_Proctor0 -
Duplicate page content errors for Web App Login
Hi There I have 6 duplicate content errors, but they are for the WebApp login from our website. I have put a Noindex on the Sitemap to stop google from indexing them to see if that would work. But it didn't. These links as far as I can see are not even on the website www.skemaz.net, but are links beyond the website and on the Web App itself eg : <colgroup><col width="529"></colgroup>
Intermediate & Advanced SEO | | Skemazer
| http://login.skemaz.net |
| http://login.skemaz.net/LogIn?ReturnUrl=%2Fchangepassword |
| http://login.skemaz.net/Login |
| http://login.skemaz.net/LogIn?ReturnUrl=%2FHome | Any suggestions would be greatly appreciated. Kind regards Sarah0 -
Canonicle & rel=NOINDEX used on the same page?
I have a real estate company: www.company.com with approximately 400 agents. When an agent gets hired we allow them to pick a URL which we then register and manage. For example: www.AGENT1.com We then take this agent domain and 301 redirect it to a subdomain of our main site. For example
Intermediate & Advanced SEO | | EasyStreet
Agent1.com 301’s to agent1.company.com We have each page on the agent subdomain canonicled back to the corresponding page on www.company.com
For example: agent1.company.com canonicles to www.company.com What happened is that google indexed many URLS on the subdomains, and it seemed like Google ignored the canonical in many cases. Although these URLS were being crawled and indexed by google, I never noticed any of them rank in the results. My theory is that Google crawled the subdomain first, indexed the page, and then later Google crawled the main URL. At that point in time, the two pages actually looked quite different from one another so Google did not recognize/honor the canonical. For example:
Agent1.company.com/category1 gets crawled on day 1
Company.com/category1 gets crawled 5 days later The content (recently listed properties for sale) on these category pages changes every day. If Google crawled the pages (both the subdomain and the main domain) on the same day, the content on the subdomain and the main domain would look identical. If the urls are crawled on different days, the content will not match. We had some major issues (duplicate content and site speed) on our www.company.com site that needed immediate attention. We knew we had an issue with the agent subdomains and decided to block the crawling of the subdomains in the robot.txt file until we got the main site “fixed”. We have seen a small decrease in organic traffic from google to our main site since blocking the crawling of the subdomains. Whereas with Bing our traffic has dropped almost 80%. After a couple months, we have now got our main site mostly “fixed” and I want to figure out how to handle the subdomains in order to regain the lost organic traffic. My theory is that these subdomains have a some link juice that is basically being wasted with the implementation of the robots.txt file on the subdomains. Here is my question
If we put a ROBOTS rel=NOINDEX on all pages of the subdomains and leave the canonical (to the corresponding page of the company site) in place on each of those pages, will link juice flow to the canonical version? Basically I want the link juice from the subdomains to pass to our main site but do not want the pages to be competing for a spot in the search results with our main site. Another thought I had was to place the NOIndex tag only on the category pages (the ones that seem to change every day) and leave it off the product (property detail pages, pages that rarely ever change). Thank you in advance for any insight.0 -
Should you bother with an "impact links" manual action
I have a couple sites that have these, and I have done a lot of work to get them removed, but there seems to be very little if any benefit from doing this. In fact, sites were we have done nothing after these penalties seem to be doing better than ones where we have done link removal and the reconsideration request. Google says "I_f you don’t control the links pointing to your site, no action is required on your part. From Google’s perspective, the links already won’t count in ranking. However, if possible, you may wish to remove any artificial links to your site and, if you’re able to get the artificial links removed, submit a reconsideration request__. If we determine that the links to your site are no longer in violation of our guidelines, we’ll revoke the manual action._" I would guess a lot of people with this penalty don't even know they have it, and it sounds like leaving it alone really doesn't hurt your site. If seems to me that just simply ignoring this and building better links and higher quality content should help improve your site rankings vs. worrying about trying to get all these links removed/disavowed. What are your thoughts? Is it worth trying to get this manual action removed?
Intermediate & Advanced SEO | | netviper0 -
Bad use of the Rel="canonical" tag
Google is currently ranking my category page instead of our homepage for our key term and we would rather have our homepage rank for the term. Would it be a bad idea to rel="canonical" our category page to our homepage? Our homepage is optimized to rank for the keyword and has more PR than our category page. However, I don't really know if this will have negative repercussions. Thanks, Jason
Intermediate & Advanced SEO | | Jason_3420 -
Does "Noindex" lead to Loss of Link Equity?
Our company has two websites with about 8,000 duplicate articles between them. Yep, 8,000 articles were posted on both sites over the past few years. This is the definition of cross-domain duplicate content. Plan A is to set all of the articles to "noindex,follow" on the site that we care less about (site B). We are not redirecting since we want to keep the content on that site for on-site traffic to discover. If we do set them to "noindex," my concern is that we'll lose massive amounts of link equity acquired over time...and thus lose domain authority...thus overall site rankability. Does Google treat pages changed to "noindex" the same as 404 pages? If so, then I imagine we would lose massive link equity. Plan B is to just wait it out since we're migrating site B to site A in 6-9 months, and hope that our more important site (site A) doesn't get a Panda penalty in the meantime. Thoughts on the better plan?
Intermediate & Advanced SEO | | M_D_Golden_Peak0 -
Rel="external" What affect if any does this have on SEO
When building Anchor text links what affect if any does rel="external" have on inlinks placed to your site. Thanks, Kjay
Intermediate & Advanced SEO | | SEOKeith0 -
Canonical URL's - Do they need to be on the "pointed at" page?
My understanding is that they are only required on the "pointing pages" however I've recently heard otherwise.
Intermediate & Advanced SEO | | DPSSeomonkey0