Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirect and Redirect Error in Moz Crawl
Hello, We have a wordpress blog attached to our magento website located at domain.co.uk/blog/ Moz was coming back showing we had multiple page versions on show (http and https) So i updated the htaccess file to what is below. This has fixed most of the errors, however the homepage is being a little tricky. Moz is now saying that the page is redirecting and redirecting again http://www.domain.co.uk/blog to
On-Page Optimization | | ATP
http://www.domain.co.uk/blog/ to
https://www.domain.co.uk/blog/ BEGIN WordPress <ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /blog/</ifmodule> RewriteCond %{HTTPS} !=on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301] RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /blog/index.php [L] END WordPress Within wordpress settings the urls are set up as follows Wordpress Address URL: https://www.domain.co.uk/blog Site Address URL: https://www.domain.co.uk/blog i tried to add a trailing / to these but it gets automatically removed. So i am assuming that wordpress is serving up https://www.domain.co.uk/blog **RewriteBase /blog/ **is re-directing it to / then my https rewrite is re-directing it again I am not sure where exactly to fix this, could anybody advise? Many thanks,0 -
Crawl Report shows Internal links as zero
We have recently updated our website (https://www.payumoney.com/) and created new pages like "https://www.payumoney.com/free-payment-gateway-India.html". we have internally linked all the pages, but when I check the crawl report from MOZ it shows internal links as 0 and these pages have the PA of 1 and not increasing. I am unable to find out the reason why is it happening?
On-Page Optimization | | payu_india0 -
How often is your domain authority updated?
I can't seem to figure out how often our domain authority is updated - it seems random, do you know typically when this happens? Thanks!
On-Page Optimization | | regineraab0 -
Is the HTML content inside an image slideshow of a website crawled by Google?
I am building a website for a client and i am in a dilemma whether to go for an image slideshow with HTML content on the slides or go for a static full size image on the homepage. My concern is that HTML content on the slideshow may not get crawled by Google and hence may not be SEO friendly.
On-Page Optimization | | aravinn0 -
Google Crawl Errors from vbseo change
We have vbseo setup on our site and for some reason a setting was changed unexpectedly and was un-noticed where it changed the URL of all the pages and so none of our pages were getting indexed by google any longer due to 401 errors. Most of our SE traffic fell off. We discovered the issue a couple weeks ago and we changed the setting back so that the URLs are the same as they were originally before but in Google webmasters it's still showing crawl errors and our search engine traffic hasn't recovered at all. We have sitemaps being sent daily.
On-Page Optimization | | RudySF0 -
Number of pages crawled in dropping from 4 to 2
The report on our campaign shows that up to 2 pages are being crawled now from 4. However, our site has more pages than this. We recently inserted code to allow crawlers. what can we do to resolve this? please assist.v
On-Page Optimization | | seoworx1230 -
Does anyone know an "updated" all in one cheat cheat for on page SEO
howdy guys, after panda, and penguin, i wanted to know if anyone has any good references for a cheat sheet that has ALL the on page SEO factors like title tag, image tags, H1-H6, etc, the basics, but since the zoo got loose I dont know if things have changed for on page SEO. For example what is the optimal KW density we should be using etc. i found this list : http://www.vaughns-1-pagers.com/internet/google-ranking-factors.htm but it seems to be a bit outdated, if someone can share something similar to this list but something more recent, like a few weeks old, i think that would help the community out a great bunch! thanks guys
On-Page Optimization | | david3050 -
SEOmoz crawl error
Hi, I'm getting a crawl error and it complains about there being missing meta description... But, the errors are all for non existent index files in directories that only contain pdf files and some thumbs of the front page... Just started trying to learn this stuff...! Cheers Rod
On-Page Optimization | | DrWho0