Site Crawl Error
-
In moz crawling error this message is appears:
MOST COMMON ISSUES
1Search Engine Blocked by robots.txt
Error Code 612: Error response for robots.txt
i asked help staff but they crawled again and nothing changed.
there's only robots.XML (not TXT) in root of my webpage it contains:
User-agent: *
Allow: /
Allow: /sitemap.htmanyone please help me? thank you
-
Thank you,
per your instructions i have changed robots.txt exactly like:
User-agent: *
Disallow:As you can see there's no Allow: / and Disallow line is blank
is that correct?
-
Hello, this is Abe from the Moz Help team. After our reported fix in your latest support ticket, you will see correct crawl crawl results in your next weekly update
-
Would you mind sharing a link to your site?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pages with Duplicate Content Error
Hello, the result of renewed content appeared in the scan results in my Shopify Store. But these products are unique. Why am I getting this error? Can anyone please help to explain why? screenshot-analytics.moz.com-2021.10.28-19_53_09.png
Moz Pro | | gokimedia0 -
How long is a full crawl?
It's been now over 3 days that the dashboard for one of our campaigns shows "Next Crawl in Progress!". I am not complaining about the length... but I have to agree that SEOMoz is quite addictive, and it's quite frustrating to see that everyday 🙂 Thanks
Moz Pro | | jgenesto0 -
Crawl test tool from SEOmoz - which URLs does it actually crawl?
I am using for the first time the crawl test tool from SEOmoz and I do not really understand which URLs the tool is going to crawl. First, it says "enter any subdomain" --> why can´t I do the crawl for the root domain? Second it says "we'll crawl up to 3,000 linked-to pages" --> does that mean that the tool crawls all internal links that it can find on the given domain? Thanks for your help!
Moz Pro | | Elke.GetApp0 -
After fixing errors can I re-crawl for diagnostics?
As I am fixing errors will the campaign automatically update to show where I have fixed issues?
Moz Pro | | eidna220 -
Crawl still in progress ...
Hi guys, New crawl on one of my campaigns is still in progress since November 27th, i didn't get new data since November 19th 2011 ... What should i do ?
Moz Pro | | DavidEichholtzer0 -
Pages Crawled: 250 | Limit: 250
One of my campaigns says: Pages Crawled: 250 | Limit: 250 Is this because it's new and the limit will go up to 10,000 after the crawl is complete? I have a pro account, 4 other campaigns running and should be allowed 50,000 pages in total
Moz Pro | | MirandaP0 -
Site metric tool
Is there a tool on here (or anywhere else) where you can upload a list of sites (say 500 sites) and be given things like mozrank on it?
Moz Pro | | thefresh0 -
How accurate is Open Site Explorer?
I noticed for one of my sites, a large number of links do not show up in Open Site Explorer, including some of my stronger links. That being the case, how much weight can I put on using these tools to compare sites? I'm not trying to bash here, I really like the tools but if my PA is 29 and my competition is 34, how much weight can I put on these numbers? Or if it says my site has 50 links from 25 domains and my competition has 60 links from 30 domains, these numbers obviously aren't very accurate? So how much weight can I put on these comparisons? Are there better tools?
Moz Pro | | MattMaresca0