What to do about "blocked by meta-robots"?
-
The crawl report tells me "Notices are interesting facts about your pages we found while crawling". One of these interesting facts is that my blog archives are "blocked by meta robots".
Articles are not blocked, just the archives.
What is a "meta" robot?
I think its just normal (since the article need only be crawled once) but want a second opinion. Should I care about this?
-
Meta robots refers to the < meta name="robots" > tag at the page header level. This is usually the case when a blog is set up with an SEO program like All In One SEO for example, where you can manually set which content is blocked. It's common to block archives, tags, and other sections, in the theory that allowing these to be crawled could either cause duplicate content issues, or drain link value from the primary category navigation.
-
In general, there are two ways you can block crawlers from indexing your content.
-
You can add a Disallow entry to your robots.txt file
-
You can add a meta tag to your pages:
What you are saying in either case is "please do not list this content in your search engine".
In general, you would not want to block your archives. There certainly can be specific cases where you only want the public to see your most current content, in which case you can block it.
-
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Product Subdomain Outranking "Marketing" Domains
Hello, Moz community! I have been puzzling about what to do for a client. Here is the challenge. The client's "product"/welcome page lives at www.client.com this page allows the visitor to select the country/informational site they want OR to login to their subdomain/install of the product. Google is choosing this www.client.com url as the main result for client brand searches. In a perfect world, for searchers in the US, we would get served the client's US version of the information/marketing site which lives at https://client.com/us, and so on for other country level content (also living in a directory for that country) It's a brand new client, we've done geo-targeting within the search console, and I'm kind of scared to rock the boat by de-indexing this www.client.com welcome screen. Any thoughts, ideas, potential solutions are so appreciated. THANKS! Thank you!
Technical SEO | | SimpleSearch0 -
Meta Titles and Meta Descriptions are not Indexing in Google
Hello Every one, I have a Wordpress website in which i installed All in SEO plugin and wrote meta titles and descriptions for each and every page and posts and submitted website to index. But after Google crawl the Meta Titles and Descriptions shown by Google are something different that are not found in Content. Even i verified the Cached version of the website and gone through Source code that crawled at that moment. the meta title which i have written is present there. Apart from this, the same URL's are displaying perfect meta titles and descriptions which i wrote in Yahoo and Bing Search Engines. Can anyone explain me how to resolve this issue. Website URL: thenewyou (dot) in Regards,
Technical SEO | | SatishSEOSiren0 -
Link rel="prev" AND canonical
Hi guys, When you have several tabs on your website with products, you can most likely navigate to page 2, 3, 4 etc...
Technical SEO | | AdenaSEO
You can add the link rel="prev" and link rel="next" tags to make sure that 1 page get's indexed / ranked by Google. am I correct? However this still means that all the pages can get indexed, right? For example a webshop makes use of the link rel="prev" and ="next" tags. In the Google results page though, all the seperate tabs pages are still visible/indexed..
http://www.domain.nl/watches/?tab=1
http://www.domain.nl/watches/?tab=24
http://www.domain.nl/watches/?tab=19
etc..... Can we prevent this, and make sure only the main page get's indexed and ranked, by adding a canonical link on every 'tab page' to the main page --> www.domain.nl/watches/ I hope I explained it well and I'm looking forward to hearing from you. Regards, Tom1 -
Will I still get Duplicate Meta Data Errors with the correct use of the rel="next" and rel="prev" tags?
Hi Guys, One of our sites has an extensive number category page lsitings, so we implemented the rel="next" and rel="prev" tags for these pages (as suggested by Google below), However, we still see duplicate meta data errors in SEOMoz crawl reports and also in Google webmaster tools. Does the SEOMoz crawl tool test for the correct use of rel="next" and "prev" tags and not list meta data errors, if the tags are correctly implemented? Or, is it necessary to still use unique meta titles and meta descriptions on every page, even though we are using the rel="next" and "prev" tags, as recommended by Google? Thanks, George Implementing rel=”next” and rel=”prev” If you prefer option 3 (above) for your site, let’s get started! Let’s say you have content paginated into the URLs: http://www.example.com/article?story=abc&page=1
Technical SEO | | gkgrant
http://www.example.com/article?story=abc&page=2
http://www.example.com/article?story=abc&page=3
http://www.example.com/article?story=abc&page=4 On the first page, http://www.example.com/article?story=abc&page=1, you’d include in the section: On the second page, http://www.example.com/article?story=abc&page=2: On the third page, http://www.example.com/article?story=abc&page=3: And on the last page, http://www.example.com/article?story=abc&page=4: A few points to mention: The first page only contains rel=”next” and no rel=”prev” markup. Pages two to the second-to-last page should be doubly-linked with both rel=”next” and rel=”prev” markup. The last page only contains markup for rel=”prev”, not rel=”next”. rel=”next” and rel=”prev” values can be either relative or absolute URLs (as allowed by the tag). And, if you include a <base> link in your document, relative paths will resolve according to the base URL. rel=”next” and rel=”prev” only need to be declared within the section, not within the document . We allow rel=”previous” as a syntactic variant of rel=”prev” links. rel="next" and rel="previous" on the one hand and rel="canonical" on the other constitute independent concepts. Both declarations can be included in the same page. For example, http://www.example.com/article?story=abc&page=2&sessionid=123 may contain: rel=”prev” and rel=”next” act as hints to Google, not absolute directives. When implemented incorrectly, such as omitting an expected rel="prev" or rel="next" designation in the series, we'll continue to index the page(s), and rely on our own heuristics to understand your content.0 -
Domain "Forwarded"?
Hi SEOMoz! The company I work for has a website, www.accupos.com, but they also have an old domain which is not used anymore called http://accuposretail.com/ These two sites had duplicate content so I requested the OLD site (http://accuposretail.com/) be redirected to accupos.com to eliminate the dupe content. Unfortunately, I do not understand completely what happened but when they performed this forwarding the accuposretail.com URL is still in use. Now it just displays EXACTLY what accupos.com displays and not something similar. The tech team told me it is forwarded but I can't help but see the URL still in the search box on top. Is this unacceptable? The actual URL has to forward and change to the accupos.com URL in order to not be duplicate content, correct? I have limited experience in this. Please let me know if we are good to go, or if I need to tell them more action is required. Thanks! Derek M
Technical SEO | | DerekM880 -
Unknown "/" added causing 404 error
I have four 404 url redirect errors that I cannot sort out. It tells me the referring url: | www.homedestination.com/calculator-mortgage-resources.html has a "/" on the end. cannot find: | www.homedestination.com/calculator-mortgage-resources.html | I cannot figure out where this referring url is; as it is in the root file without a "/" on the end. Could it be on a page somewhere? All my Dreamweaver page link tests come back ok. I must be missing something simple and would value help for others who may spot it? Thanks! |
Technical SEO | | jessential0 -
Does Google follow links in "id" tag?
Hello, For functionality purposes, I need to wrap separate blocks of content with a tag. The main question is whether Google will follow this URL, even though it is not a hyperlink on the page, just a URL used for functionality purposes. We will have 10-20 of these types of span tags with a different URL for each one, and we just want to be sure that Google will not be following these URLs that are not links. Thanks!
Technical SEO | | Hakkasan0 -
I have both a ".net" and a ".com" address for the Same Website.....
I have mysite.net and mysite.com......They are both the same age, however, we always had it so that the mysite.com address forwarded to the mysite.net address. The mysite.net address was our main address forever. We recently reversed that and made the mysite.com address the main address and just have mysite.net forward to the mysite.com address. I'm wondering if this change will affect our rankings since a lot of the backlinks we've acquired are actually pointing to mysite.net and not mysite.com (our new main address)???
Technical SEO | | B24Group0