Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
403 Forbidden Crawl report
Hi, I am getting 403 forbidden crawl report on some of my pages. However the pages are loading fine. Also when asked my web developer told that some times reports show errors when there is nothing wrong. Also will the errors affect the SEO/Ranking etc.
On-Page Optimization | | ghrisa65
Some of the links:
https://www.medistaff24.co.uk/contact-us/ https://www.medistaff24.co.uk/elderly-care-in-evesham-worcestershire/ https://www.medistaff24.co.uk/hourly-home-care-in-evesham/0 -
To change or not to change. DO we update our HTTP page to HTTPS? especially those with basic forms (phone, email, name)??
I recently went to a conference where a speaker strongly urged us to migrate to HTTPS before January 2017. I don't see any other sites referencing to make the switch before January 2017. whats the deal? 😉
On-Page Optimization | | millenniumsi0 -
PDF Instructions come up in Crawl report as Duplicate Content
Hello, My ecommerce site has many PDF instruction pages that are being marked as duplicate content in the site crawl. Each page has a different title, and then a PDF displayed in an iframe with a link back to the previous page & to the category that the product is placed in. Should I add text to the pages to help differentiate them? I included a screenshot of the code that is on all the pages. Thanks! Justin 9tD9HMr
On-Page Optimization | | JustinBSLW0 -
Massive increase in Moz crawl.
I have a subdomain which has just started to be crawled by Moz, Previously this wasn't the case. The sub-domain had 16,000+ issues. Why has Moz started to count sub-domains as part of the main domain, has Google started to do this aswell?
On-Page Optimization | | danwebman0 -
Can I force an update of Grade Reports?
It looks like my weekly crawl has finished, but my Grade Reports still reflect last week. Is there a way to manually update them, or do I just have to wait it out?
On-Page Optimization | | FDAitsupport0 -
How does a keyword get crawled and pointed at a certain page
I was wondering if you can give me some insight on how a keyword that I put on my campaign gets linked to a specific URL on my website by SEOMoz or Google. For example: updating a brick fireplace is my keyword. On the campaign when I am looking at my on page optimization, the URL assigned (or given) to it is my homepage. How is this determined and is there a way around it and or directing it to the correct page? Thanks
On-Page Optimization | | SammyT0 -
How to force a refresh after on-page optimisation update
After updating areas highlighted in the On-Page Optimization report even after clicking the [Grade My On-page Optimization] the results don't refresh or reflect the changes eg The h1 tag does include the exact search term and there is bolded examples of the keyword phrase but report says not! Is there a way to force an update or is it a time related issue?
On-Page Optimization | | RobWillox0 -
I built a website on magentogo - IrisScottPrints.com. The seomoz crawl report states 301 rel canonical crawl notices. What if anything should I change?
Wondering if I should remove "IRIS SCOTT PRINTS |" from all the title tags and/or change the url structure of the pages, to not include the breadcrumbs... I don't really understand the whole rel canonical structure thing. Also lots of errors on page title too long - does that really matter? Lots of faith in everyone here. Thanks in advance. Marcia
On-Page Optimization | | RedTrout0