Moz crawling bot
-
Hi guys,
in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".
I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).
However, my robots.txt doesn't have such entry. Take a look:
Automatically banned scanners and crawlers section
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
Section end
What could be the problem here, then? Why does the Moz bot think I'm blocking it?
-
Thank you Keri, problem solved.
-
If you pull out dotbot from your exclusions, you should be fine.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Solved Moz Link Explorer slow to find external links
I have a site with 48 linking domains and 200 total links showing in Google Search Console. These are legit and good quality links. Since creating a campaign 2 months ago, Moz link explorer for the same site only shows me 2 linking domains and 3 total links. I realise Moz cannot crawl with the same speed and depth as Google but this is poor performance for a premium product and doesn't remotely reflect the link profile of the domain. Is there a way to submit a sitemap or list of links to Moz for the purpose of crawling and adding to Link Explorer?
Link Explorer | | mathewphotohound0 -
Google Disavow File Format and MOZ Spam Score Updates
Hi, Is there a defined file format for Google disavow file name? Does it has to be disavowlinks.txt or can we do this like domain-name-date.txt ? Also, since Google does not share their data with Moz, how does MOz updates its spam score after we disavow the bad links? Do we need to connect Google search console with Moz?
Link Explorer | | Sunil-Gupta1 -
Moz Spam Score 9/17 when there are no links
Hi I was just looking at a competitors new site in OSE and it had a DA and PA of 1 which I would expect and also no links, which again I would expect, basically the guy used to design stuff for me and thinks he can do a better job so he is giving it ago. But he has a spam score of 9/17 but no links - how is this even possible. Thanks Andy
Link Explorer | | Andy-Halliday0 -
Sudden Spike in 404 Pages Not Found in Moz Crawl But No Errors in WMT
Recently I received a spike in errors from the Moz crawler. When I looked into the matter I noticed that all the URI's looked right but then I looked a little closer and there was a /page/2 and /page/3 in front of the URI's. I'm running a WordPress website. Immediately I thought to myself this must be some kind of caching or permalinks error. So I disabled all my plugins including W3 Total Cache and ran the Integrity Link Crawler for the Mac and found that the errors were still popping up. 404-errors-ncworkercomp.png?dl=0
Link Explorer | | NCCompLawyer0 -
Only two pages crawled
Hi community, I'm having trouble viewing data for all of our pages. Moz is only crawling two of our pages, and we have a lot more than that. Any suggestions? Thanks, Emily
Link Explorer | | SkuidEmily0 -
Learn how to use Moz's Link Intersect tool to build your link profile. Get your Daily SEO Fix!
The quantity and (more importantly) quality of backlinks to your website make up your link profile; one of the most important elements in SEO and an incredibly important factor in search engine rankings. In today's Daily SEO Fix, How to Use Link Intersect to Build Your Link Profile, Tori shows you how to use Moz's Link Intersect tool to analyze the competitions' backlinks plus, find opportunities to build links and strengthen your own link profile. This video is one of our last videos in The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, make sure to visit the Daily SEO Fix channel on YouTube. And, if you have suggestions for future tutorials, please let us know via the comments below. Thanks!
Link Explorer | | kellyjcoop2 -
SEO Benefit from Moz Profile link?
Hi all Excuse the rookie question, but do the website links in Moz profile give positive SEO benefits? If so how long should it take to start seeing improvements or have the linking domain in Open explorer and is there a way to speed things up? Thanks in advance
Link Explorer | | IsaCleanse0 -
Does OSE crawl our site often? Or do we have an other problem? (internal links)
I am wondering if i can find out when OpenSiteExplorer crawled our entire website for the last time.
Link Explorer | | wilcoXXL
A few months ago we added some internal textlinks to all of our productpages. Those are links to the brandpage of that particular product. Some of our brands do have up to 1500 products, so there should be at least 1500 internal links pointing to that brandpage. But OSE still gives me a wrong count. It says there is only 1 internal link pointing to that brandpage.
Is it because OSE never crawled our site again? Or am i missing something here?(maybe too many internal links to one specific brandpage is not okay?) I'll hope you guys can help me out with this one.0