Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why will my page description not update??!!
I updated my page title and description along time ago, the title updated but not the description. I updated it in google search console, but it keeps pulling the first few lines from the page content instead, which isn't really an attention grabbing sentence?? Any ideas? Here's the page http://thumannagency.com/personal-insurance/home-insurance This is what my Title/ description comes up as... Homeowners Insurance Quotes Dallas TX | Thumann Agency <cite class="_Rm">thumannagency.com/personal-insurance/home-insurance</cite>When shopping for homeowners insurance in Dallas, TX our local agents will provide you with home insurance quotes from highly rated home insurance ...
On-Page Optimization | | MissThumann0 -
Update old article or publish new content and redirect old post?
Hi all, I'm targetting a keyword and we used to rank quite good for it. Last couple of months traffic of that keyword (and variations) is going down a bit. I wrote an extensive new post on the same topic, much more in dept and from 600 to 1800 words covering the same topic. Is it better to update the old article and mention that it's updated recently, or publish a new post and redirect the old post to the new post?
On-Page Optimization | | jorisbrabants0 -
Site update and what to do about current keyword rank
Hi im in the process of giving my site a major update as its only one page and not responsive . The the new site will have a homepage with a list of my services with a small description of each and each service will link to its own page. My one page site at the moment ranks number one in my area for a low competition keyword (moz keyword score of 13%) of course this sends me very little traffic. With my new site im also going for more keywords but these will be Moz KWD 30% and 40%. I know this will be quite hard and take some time but i'm pretty sure I know what I need to do to get there. Now my question is what do i do with my current home page (only page) that ranks for that low comp keyword?
On-Page Optimization | | juun
I dont want to lose my rank for that, so do i make a new page on my new site that is optimized for that keyword, but then I don't want to 301 my homepage to that so I guess as its such an easy word to rank for my new page would soon rank for that? What are your thoughts and advice please, thanks in advance.1 -
Moz Crawl Shows Duplicate Content Which Doesn't Seem To Appear In Google?
Morning All, First post, be gentle! So I had Moz crawl our website with 2500 high priority issues of duplicate content, not good. However if I just do a simple site:www.myurl.com in Google, I cannot see these duplicate pages....very odd. Here is an example....
On-Page Optimization | | scottiedog
http://goo.gl/GXTE0I
http://goo.gl/dcAqdU So the same page has a different URL, Moz brings this up as an issue, I would agree with that. However if I google both URL's in Google, they will both bring up the same page but with the original URL of http://goo.gl/zDzI7j ...in other words, two different URL's bring up the same indexed page in Google....weird I thought about using a wildcard in the robots.txt to disallow these duplicate pages with poor URL's....something like.... Disallow: /*display.php?product_id However, I read various posts that it might not help our issues? Don't want to make things worse. On another note, my colleague paid for a "SEO service" and they just dumped 1000's of back-links to our website, of course that's come back to bite us in the behind. Anyone have any recommendations for a good service to remove these back-links? Thanks in advance!!0 -
Ok to ignore Overly-Dynamic URL from Moz crawl?
I am developing an ecommerce site, just ran it through the Moz crawl to see what's what and it has come back with a lot of issues. Most of these issues are around duplicate page titles (it is not happy with paginated titles, ie Shoes, Shoes Page 2, Shoes Page 3 etc) and it has also found a lot of Overly-Dynamic URL's. Again, these seem to be from some of the search functions and filters used Accessories&pto_sort=priceAsc&pto_page=6 other than spending a lot of time and effort trying to rewrite these urls there is little I can do about them. Should I just ignore this? I wouldn't imagine it having a massive impact on the rankings of the pages. Thanks, Carl
On-Page Optimization | | GrumpyCarl0 -
On Page Optimization not updating after SSL change
Last week we changed our URLs to be only HTTPS (they were a mixture of both for a couple of weeks). We did this by adding a 301 redirect from HTTP to HTTPS. However, the On Page Optimization is now showing the same URL/Keyword twice for a few of our pages. One has a grade A, one has a grade B. The grade B complains about "Rel Canonical" not matching, but the other doesn't. This is pretty confusing - should it not have crawled the pages and noticed the 301 redirects and combined the entries or removed the HTTP ones?
On-Page Optimization | | RitchieF19800 -
SEOmoz crawl error
Hi, I'm getting a crawl error and it complains about there being missing meta description... But, the errors are all for non existent index files in directories that only contain pdf files and some thumbs of the front page... Just started trying to learn this stuff...! Cheers Rod
On-Page Optimization | | DrWho0 -
Google webmaster tools data update frequency?
What is the lag time for changes to a site to be reflected in Google webmaster tools Diagnostics section? They pointed out some duplicate titles which I fixed a week ago and yet they still show up as an HTML Suggestion. What has your experience been with making changes and then seeing them reflected in the HTML Suggestions section? My site is crawled every day, including the pages I have updated with new titles. Seems like it takes a while for the data to trickle into Webmaster tools, no?
On-Page Optimization | | scanlin0