Rogerbot's crawl behaviour vs google spiders and other crawlers - disparate results have me confused.
-
I'm curious as to how accurately rogerbot replicates google's searchbot
I've currently got a site which is reporting over 200 pages of duplicate/titles content in moz tools. The pages in question are all session IDs and have been blocked in the robot.txt (about 3 weeks ago), however the errors are still appearing.
I've also crawled the page using screaming frog SEO spider. According to Screaming Frog, the offending pages have been blocked and are not being crawled. Webmaster tools is also reporting no crawl errors.
Is there something I'm missing here? Why would I receive such different results. Which one's should I trust? Does rogerbot ignore robot.txt? Any suggestions would be appreciated.
-
Thanks for your response. I was beginning to think this question had been left to rot.
I'm not getting any errors in WMT. What is concerning is that Roger is returning almost 300 errors of dupe content, which is obviously a problem. Screaming frog is no longer finding the pages (they've been blocked in the robot.txt) I guess what I'm trying to ask here is how can I be sure that my dupe content has been effectively blocked from google's spider.
Is there anyway to check?
Thanks for your help.
-
I've see similar concerns from others, it seems "rogerbot" does ignore certain things that other bots consider.
Don't worry about it, if it's not being flagged in WMT it shouldn't be an issue.
Take Roger as a guide rather than an iron fist bot like googlebot.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I'VE DONE EVERYTHING RIGHT BUT STILL GET LOW GOOGLE RANKING
Moz Pro Dashboard compared my site to three competitors' site. My site is better in every aspect. My DA is higher, I have four times more external links, I have more relevant content than my competition, my site also is the fastest, I have solved all redirect issues, there is no broken link internally or externally. We are custom home builders. So all our sites have numerous full screen images. I have optimized all of them. When I ran WebPageTest on my competition, they all fail. My site got A's in all categories. The three competitors' sites rank first, second and third. My site ranks bottom on the 2nd page. Since my site was online last January, I have spent thousands of dollars on SEO work. Its url is www.pokudesign.com I am hoping somebody can point me in the right direction. I am attaching a screen shot of my Moz Pro Dashboard. frEpZ
Moz Pro | | pokongku0 -
Need help understanding search filter URL's and meta tags
Good afternoon Mozzers, One of our clients is a real estate agent and on that site there is a search field that will allow a person to search by filtered categories. Currently, the URL structure makes a new URL for each filter option and in my Moz reports I get the report that there is missing meta data. However, the page is the same the filter options are different so I am at a loss as to how to proper tag our site to optimize those URL's. Can I rel canonical the URL's or alt rel them? I have been looking for a solution for a few days now and like I said I am at a loss of how to properly resolve these warning messages, or if I should even be concerned with the warning messages from Moz (obviously I should be concerned, they are warning messages for a reason). Thank you for your assistance in advance!
Moz Pro | | Highline_Ideas0 -
Get into Google : New Sites
I have a brand new website. It was created 10 days ago. How long would it take for it to show up in search results? I understand that since the site is new, there are no sites sending it backlinks. Also, i have optimized the page for my keyword "xyz" and it received an A grade. The site does not figure even in the top 50 results. Please help me out. It is a one page web application that needs to drive traffic to survive.
Moz Pro | | dl_s0 -
Facebook shared vs liked
Hi, looking at the "Open Site Explorer" I see : | Times Shared on Facebook: | 12 | 25 | 7 | 8 | -- |
Moz Pro | | KJMM
| Facebook Likes | 0 | 4 | 5 | 9 | -- | But what is the differencebetween Times Shared on Facebook and Facebook Likes??0 -
Local search results reporting
Is there a way I can report on local search results for our keywords by city?
Moz Pro | | RPM5
For example could I specify Phoenix or Chicago and then see how we rank as if we did the search in that city?0 -
SEOMoz PR vs. Google PR
I have the SEO MozBar set up and for one of the sites we are maintaining, it is showing them on a 59 as PA in the MozBar. When I check it on Google PR, it has either no info or a 0. http://www.prchecker.info was one of the ones I found which shows them as 0. Any help would be greatly appreciated.
Moz Pro | | Champions0 -
Why would PA be 1 (0 links from 0 root domains) if it's linked to internally?
Question just about said it all: I've seen a number of pages on sites that have a PA of 1 (with the metrics being 0 links from 0 root domains) when I can see on the site that it is linked to internally - from the main nav (which is CSS, not Javascript) and also from the footer, if not other places. Why would this be? Update: upon looking further at the site, it appears that there's some kind of redirect going on, where the page linked to from the nav actually redirects to the real page. Would that eliminate PA, even if it's a 301? And additionally, is whatever is causing this lack of PA a reflection of how Google would relate to the page? Thanks, Aviva
Moz Pro | | debi_zyx0