Our crawler was not able to access the robots.txt file on your site
-
I've submitted my website to be crawled by Moz and done everything I can according to the troubleshooting guides. Please help!
-
Disabled my CDN and it worked!
Thanks so much!
-
Hi - Thanks for reaching out.
When I tried to curl your robots file from our user-agent string, I was redirected 50 times (then timed out as max redirects). My guess would be you have some type of bot protection with Cloudflare. You can reach out to your hosting admin and have them locate "rogerbot" in the logs. They will need to most likely whitelist our crawler to access the site.
If you need further assistance with this, please reach out to [email protected] directly, and we can assist!
Thanks!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this. I assume it would be: User-agent: Moz
Getting Started | | Simon-Plan
Disallow: / But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text! Many thanks.0 -
Can't Crawl Site - but deducting crawls.
Why am I being deducted crawls if MOZ keeps telling me that it can't crawl my site?
Getting Started | | BloggyMoms1 -
Site with 2 domains - 1 domain SEO opimised & 1 is not. How best to handle crawlers?
Situation: I have a dual domain site:
Getting Started | | DGAU
Domain 1 - www.domain.com is SEO optimised with product pages and should of course be indexed.
Domain 2 - secure.domain.com is not SEO optimised and simply has checkout and payment gateway pages. I've discovered that Moz automatically crawls Domain 2 - the secure.domain.com site and consequently picks up hundreds of errors.
I have put an end to this by adding a robots.txt to stop rogerbot and dotbot (mozs crawlers) from crawling domain 2. This fixes my errors in Moz reports however after doing more research into 'Crawler Control' I figure this might be the best option. My Question: Instead of using robots.txt to stop moz from crawing all of Domain 2 should I use on each page of domain 2? I believe this would then allow moz and google to crawl Domain 2 but also tell them both not to index it.
My understanding is that this would be best, and might even help my overall SEO by telling google not to give any SEO value to the Domain 2 pages?0 -
Our crawler was not able to access the robots.txt file on your site
Hello Mozzers! I've received an error message saying the site can't be crawled because Moz is unable to access the robots.txt. I've spoken to the webmaster and he can't understand why the robot.txt can't be accessed in Moz. https://www.thefurnshop.co.uk/robots.txt and Google isn't flagging anything up to us. Does anyone know how to solve this problem? Thanks
Getting Started | | tigersohelll0 -
Open site explorer
Excuse my ignorance but I am very new to all this. I have a new site www.sassandgrace.co.uk and am using MOZ to try and get it right first time. On OSE I am seeing different results for http://www.sassandgrace.co.uk/ and http://www.sassandgrace.co.uk i.e with an without / ) and different results again for http://sassandgrace.co.uk These differences relate to the social page metrics but I would be keen to get one consistent reading. What do I need to do? Also, I know that I have links to the site but none are showing at all. On google console there are a couple showing but I know that there are many more. Is it just a matter of time before they are 'found' or is there something that I should be doing? Sorry for what are probably very basic questions but any help is appreciated.
Getting Started | | Sassandgrace0 -
Does the Moz crawler have a static ip address?
We block AWS from crawling our site. Does the moz crawler use a static ip address that we could allow? I'm currently not able to add a campaign because Moz can't connect to our site.
Getting Started | | uShip0 -
Why cant I add my site to a campaign?
I am trying to add my site www.dominickdalsanto.com to a campaign and it keeps telling me the URL is invalid. I have tried entering it as: dominickdalsanto.com dominickdalsanto.com/ www.dominickdalsanto.com http://www.dominickdalsanto.com Nothing works. I even tried the redirect domains I have ayres-seo.com and still nothing. I tried a few of my other sites too and it works for only one of them. It also would not take dominickinargentina.com Can someone help me please? Thanks!
Getting Started | | Ayres-SEO0 -
New to using MOZ. Familiar with Google Analytics. With MOZ is there a code snippet to include on my site?
Just taken over web marketing responsibilities at my company. Will be doing some major website upgrades soon. I'm not familiar with MOZ and don't want to overwrite anything. So when setting up MOZ, is there a code snippet that goes anyplace on the site like there is with Google analytics? Thanks.
Getting Started | | NanoLumens0