Rogerbot's crawl behaviour vs google spiders and other crawlers - disparate results have me confused.
-
I'm curious as to how accurately rogerbot replicates google's searchbot
I've currently got a site which is reporting over 200 pages of duplicate/titles content in moz tools. The pages in question are all session IDs and have been blocked in the robot.txt (about 3 weeks ago), however the errors are still appearing.
I've also crawled the page using screaming frog SEO spider. According to Screaming Frog, the offending pages have been blocked and are not being crawled. Webmaster tools is also reporting no crawl errors.
Is there something I'm missing here? Why would I receive such different results. Which one's should I trust? Does rogerbot ignore robot.txt? Any suggestions would be appreciated.
-
Thanks for your response. I was beginning to think this question had been left to rot.
I'm not getting any errors in WMT. What is concerning is that Roger is returning almost 300 errors of dupe content, which is obviously a problem. Screaming frog is no longer finding the pages (they've been blocked in the robot.txt) I guess what I'm trying to ask here is how can I be sure that my dupe content has been effectively blocked from google's spider.
Is there anyway to check?
Thanks for your help.
-
I've see similar concerns from others, it seems "rogerbot" does ignore certain things that other bots consider.
Don't worry about it, if it's not being flagged in WMT it shouldn't be an issue.
Take Roger as a guide rather than an iron fist bot like googlebot.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
403s: Are There Instances Where 403's Are Common & Acceptable?
Hey All, Both MOZ & Webmaster tools have identified 403 errors on an editorial site I work with (using Drupal CMS). I looked into the errors and the pages triggering the 403 are all articles in draft status that are not being indexed. If I am not logged into our drupal and I try to access an article in draft status I get the 403 forbidden error. Are these 403's typical for an editorial site where editors may be trying to access an article in draft status while they are not logged in? Webmaster tools is showing roughly 350 pages with the 'Access Denied' 403 status. Are these harmful to rank? Thanks!
Moz Pro | | JJLWeber1 -
Google has a penalty on my website? How to resolve?
I recently purchases Moz pro, hoping to find out why my website isn't ranking on specific keywords. I'm done analytics on myself and my competition and it seems that I rank better than them on 95% of the categories (using OpenSiteExplorer and keyword research tools). So I purchases Moz Pro hoping I'd find the glaring error in my ways. I found a few small issues I was having, and have dealt with them. Overall however, they were small issues, and have led me to believe Google as a penalty in place for my website (SEO friend have also suspected this). In my Google Webmaster tools, nothing is marked under Manual Actions. So does anyone have a way to contact Google and have them directly look at my website to insure there is no errors in their system?
Moz Pro | | ZSuttonPhoto10 -
Data Update for RogerBot
Hi, I noticed that rogerbot still give me 404 for http://www.salustore.com/capelli/nanogen-acquamatch.html refferal form http://www.salustore.com/protocollo-nanogen even I made changes since a couple of week. Same error with one "Title Element Too Short" on our site. Any suggestion on how to refresh it? Best Regards n.
Moz Pro | | nicolobottazzi0 -
Rank Tracking results
Hi, Some problem with rank tracking from research tools: I rank #1 for this keyword "w immo" for URL : http://www.wimmobiliere.com in my campaign monitor (ranking tab) In rank tracking for the same Search Engine (google.be) i am not in the 50. Do you know why? Some images that might help... Rank_1<a class="attachment-title">Rank 1</a>Rank_2<a class="attachment-title">Rank 2</a>
Moz Pro | | VOUS0 -
Google Places have been entirely replaced by new Google+
Read this! http://searchengineland.com/google-places-is-over-company-makes-google-the-center-of-gravity-for-local-search-122770 http://www.nydailynews.com/news/national/review-google-adds-zagat-reviews-local-article-1.1086985 Any ideas? How's your Google+ circle look like? What will happened to our SEO ranking? IS Rogger MOZ is ready for that change? What about SEOmoz will give us some Local/Pages/Google+ tools?
Moz Pro | | Elchanan0 -
SEOMoz rocks, but is it breaking Google's webmaster guidelines when it comes to check rankings?
Hi all, The Google's webmaster guideline in question reads... "Don't use unauthorized computer programs to submit pages, check rankings, etc. Such programs consume computing resources and violate our Terms of Service. Google does not recommend the use of products such as WebPosition Gold™ that send automatic or programmatic queries to Google." So, when WebPosition Gold is not recommended then how come SEOMoz is acceptable to check ranks for websites belonging to SEOMoz PRO? Do Google neglects breaking of few guidelines by good boys? Thanks in advance.
Moz Pro | | vinodchaudhary0 -
Errors on my Crawl Diagnostics
I have 51 errors on my Crawl Diagnostics tool.46 are 4xx Client Error.Those 4xx errors are links to products (or categories) that we are not selling them any more so there are inactive on the website but Google still have the links. How can I tell Google not to index them?. Can those errors (and warnings) could be harming my rankings (they went down from position 1 to 4 for the most important keywords) thanks,
Moz Pro | | cardif0