"Issue: Duplicate Page Content " in Crawl Diagnostics - but these pages are noindex
-
Saw an issue back in 2011 about this and I'm experiencing the same issue. http://moz.com/community/q/issue-duplicate-page-content-in-crawl-diagnostics-but-these-pages-are-noindex
We have pages that are meta-tagged as no-everything for bots but are being reported as duplicate. Any suggestions on how to exclude them from the Moz bot?
-
Technically that could be done in your robots.txt file but I wouldn't recommend that if you want Google to crawl them too. I'm not sure if Rogerbot can do that. Sorry I couldn't be more help.
If you don't get one of the staffers on here in the next few days, I would send a ticket to them for clarification.
If you decide to go with robots.txt here is a resource from Google on implementing and testing it. https://support.google.com/webmasters/answer/156449?hl=en
-
Thanks for the information on Rogerbot. I understand the difference between the bots from Google and Moz.
Some errors reported in Moz are not real. For example we use a responsive slider on the home page that generates the slides from specific pages. These pages are tagged to no-everything so as to be invisible to bots, yet they are generating errors in the reports.
Is there anyway to exclude some pages from the reports?
-
Don't forget that Rogerbot (moz's crawler) is a robot and not an index like Google. Google used robots to gather the data but the results we see is an index. Rogerbot will crawl the pages regardless of noindex or nofollow.
Here is more info on RogerBot http://moz.com/help/pro/rogerbot-crawler
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to deal with this duplicate content
Hello our websites offers prayer times in the US and UK. The problem is that we have nearby towns where the prayer times are the same and the pages (exp : https://prayer-times.us/prayer-times-lake-michigan-12258-en and https://prayer-times.us/prayer-times-lake-12147-en) are in duplicate . Same issue for this page https://prayer-time.uk/prayer-times-wallsend-411-en How can we solve this problem
On-Page Optimization | | Zakirou0 -
Does using Yoast variables for meta content overwrite any pages that already have custom meta content?
The question is about the Yoast plugin for WP sites. Let's say I have a site with 200 pages and custom meta descriptions / title tags already in place for the top 30 pages. If I use the Yoast variable tool to complete meta content for the remaining pages (and make my Moz issue tracker look happier), will that only affect the pages without custom meta descriptions or will it overwrite even the pages with the custom meta content that I want? In this situation, I do want to keep the meta content that is already in place on select pages. Thanks! Zack
On-Page Optimization | | rootandbranch0 -
How to fix thin content issue?
Hello! I've checked my website via Moz and received "thin content" issue: "Your page is considered to have "thin content" if it has less than 50 words" But I definitely know that we have 5 text blocks with unique content, each block consist of more than 50 words. Do you have any ideas what may cause this issue? Thanks in advance, Yana
On-Page Optimization | | yanamazault0 -
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
On-Page Optimization | | Jacksons_Fencing0 -
Ranking for "synonym" terms on separate pages?
(My title says "synonym" but it's not exactly the most accurate word, but works best for the title_) I have a site that ranks #1 for a term, and let’s pretend it’s “cheap phone”. It’s also ranks #1 for “cheap phone service” and #3 for “cheap phone plans”. These are all the home page with those rankings I have a sub page whose natural title would be “Cheap Phone Plans” or “Cheap Phone Service”. I have it named something these and it is not optimized for either of these terms because I think it would be best to not mess with the good rankings I have already for those two terms So here’s my question: what would likely be the outcome if I optimized that subpage for “Cheap Phone Plans” or “Cheap Phone Service”? If Google began to direct searchers of this term to my subpage rather than my home page, would my home page lose some of it’s ranking with it’s main and most popular keyword, “cheap phone? Thanks!
On-Page Optimization | | bizzer0 -
Rel="canonical" link should they be to or from an "SEO friendly" url
Thanks for taking the time to review this. So for our example, lets use the following SEO friendly link: http://hiu.calibermediagroup.com/undergraduate-on-campus/academics/colleges/pacific-christian-college-of-ministry-and-biblical-studies/BA-biblical-studies We'll call this link the SEO VERSION The title of the college is" Pacific Christian College of Minstry and Biblical Studies" The title of the program is "BA Biblical Studies" The QUERY version of the link to this page would be something like: http://hiu.calibermediagroup.com/undergraduate-on-campus/academics/colleges/index.php?collegeid=22&programid=34 Keep in mind that the meta title, description, and keyword tags for the page are all administerable The SEO VERSION is automatically created from the title of the college, and the title of the program. Each one of these titles can be overidden with a URL slug individually. For instance, the admin could make the link: http://hiu.calibermediagroup.com/undergraduate-on-campus/academics/colleges/pacific-christian-college-of-ministry/biblical-studies by changing the slug for the college to "pacific-christian-college-of-ministry" and the slug for the program to "biblical-studies". Let's call this version the SLUG VERSION So now we have multiple ways to get to the same content. The question on the table is what is best practice for the rel="canonical" link to keep from getting dinged for duplicate content. Let's say that our SEO VERSION is the canonical link for 1 year. Then the choice was made to optimize the links thru the slugs creating the SLUG VERSION. My assumption is that we would keep the SEO VERSION as the canonical link. But then let's say 6 months later that the title of the program is changed in the admin. Now the SEO VERSION has changed and so has the canonical link. Do we lose the link juice garnered over the last 18 months? It would seem to me, that if we use the QUERY version as the canonical link, then any optimizations or changes affect everything except the canonical link, thus keeping the previous link juice earned. But is having an ugly URL as the canonical link detrimental to SEO? Please advise.
On-Page Optimization | | robertdonnell0 -
Duplicate Content Indentification Tools
Does anyone have a recommendation for a good tool that can identify which elements on a page are duplicated content? I use Moz Analytics to determine which pages have the duplicated content on them, but it doesn't say which pieces of text or on-page elements are in fact considered to be duplicate. Thanks Moz Community in advance!
On-Page Optimization | | EmpireToday0 -
Duplicate content on video pages
Hi guys, We have a video section on our site containing about 50 videos, grouped by category/difficulty. On each video page except for the embedded player, a sentence or two describing the video and a list of related video links, there's pretty much nothing else. All of those appear as duplicate content by category. What should we do here? How long a description should be for those pages to appear unique for crawlers? Thanks!
On-Page Optimization | | lgrozeva0