Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the [email protected]
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Moz Pro provide a list of external (vs internal) links on our website?
Is there a tool (in MOZ?) that lists all the links on our website that point to external links (vs internal links)?
Link Explorer | | analytics-moz0 -
Moz tool to estimate a website's organic traffic?
Moz has tons of great tools, so I was quite suprised there is nothing similar to Alexa's siteinfo tool, or SimilarWeb's site comparison tool. Am I missing something? Or does such a tool exists at Moz? Or is there a different way of estimating my competitor's traffic? It would be great if Moz would have something like that.
Link Explorer | | GrandRapids0 -
Does moz pro standard give access to open site explorer & keyword explorer?
Does moz pro standard give access to open site explorer & keyword explorer?
Link Explorer | | SearchOpt1 -
Why is moz telling me I have duplicate content, but neither the content nor the urls are duplicates?
I just upgraded our website to a new one. This is the first crawl of the new website. It is telling me I have 24 Critical issues, all of which are duplicate content errors. Thing is, the urls are not duplicates and the content on the page is not duplicated either. Example, here is one error where it says there are two duplicates: <colgroup><col width="595"></colgroup>
Link Explorer | | damon1212
| http://winterguardtarps.com/portfolio-item/props-10 |
| http://winterguardtarps.com/portfolio-item/props-2 |
| http://winterguardtarps.com/portfolio-item/props-7 | There are photos in our portfolio, and none of them are the same. I'm a bit of a noob, but what am I missing here?0 -
MOZ Rank Or MOZ Wrong!?
Hi everyone, I would like to show up you a majestic mistake of ranking. I've checked some of my competitors and encountered that site has no DmT(Domain Trust Rank) but its mT(Moz Trust) is 3.83.Its good! but its not good! which one is true? is it possible that a domain isn't trust but a home page is trust!? zv99ZJV zv99ZJV
Link Explorer | | abolfazli0 -
Open Site Explorer metrics report says we have 9 linking root domains, but way more than 7 are listen. What gives?
When I run a report on my website in OSE (SelectAccount" metrics tells me we have 7 "established" root domains and 9 total links. But the detailed report below has many dozens of links. What is metrics telling me?
Link Explorer | | SelectAccount0 -
Site Mark-up is Abnormally Small
My site www.brightonsoundsystem.co.uk has been optimised for speed so I have minimised the code needed. Now if I put it through the OSE spam analysis it has a flag for "Site Mark-up is Abnormally Small". What ratio of visible text compared to mark-up code is being used to trigger this flag. Also as this is the only flag I have is ti worth the time fixing.
Link Explorer | | Brighton-Soundsystem0 -
Why MoZ Bar gives different PA and DA to my homepage? Surly they should have the same metrics?
Hi guys, is it normal that when looking at my homepage with the mozbar i get the same DA and PA? It seems that it should provide the same results. Thankss
Link Explorer | | artdivision0