How to turn off automated site crawls
-
Hi there,
Is there a way to turn off the automated site crawl feature for an individual campaign?
Thanks
-
No problem!
Thanks for the suggestion!
I will pass that on as feedback
Eli
-
Hi Eli,
Thanks for your response.
I'd really suggest that the functionality to prevent the site crawl feature is built into the campaign tool. It would take dev effort to disalow the Moz bot where I'm based.
Thanks
-
Hey!
Thanks for reaching out to us!
Moz uses a crawler called Rogerbot to crawl your site and populate the "Site Crawl" section of your Campaign. Roger isn't a search engine index, so if URLs aren't blocked by the robots.txt file, then Roger will crawl them!
If you'd like to remove pages from the crawl, you can disallow them in the robots.txt file. Here's what that exclusion would look like if you were wanting to keep him from crawling any pages on your site:
User-agent: rogerbot
Disallow: *I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl-test not doesn't finish
Hello, I have used this crawl-test on 2 website 3 days ago, and it hasn't finished yet. I'm wondering if the crawler is on an infinite loop, or has crashed without sending back an error. I could re-launch the test, but if it's really still crawling, I don't want to loose any work in progress. Is there any way to check the status of a crawl?
Moz Bar | | Nobody16116145880332 -
Crawl report shows that it gets 4xx errors for pages that work fine. Why?
On the crawl report it has all these "Critical Crawler Issues". They all say "4xx Error", yet when i click on the link from the crawler report, it goes to a perfectly functioning page, not a 404 page or anything. If i click in it actually says it's a 403 error. It's all for pages generated by the IDX solution for our real estate website. Is Moz broken or am i missing something? Here are a couple examples: <dl class="crawl-page-details-list"> <dd class="crawl-page-details-list-emphasis">https://teamvivi.com/homes-for-sale-map-search/</dd> <dd class="crawl-page-details-list-emphasis"> <dl class="crawl-page-details-list"> <dd class="crawl-page-details-list-emphasis">https://teamvivi.com/email-alerts/</dd> </dl> </dd> </dl>
Moz Bar | | TeamViviRealEstate0 -
Moz Site Crawl Test 404
Crawled site a number of times using Crawl Test. Its reporting 404's from files that are actually present. What do you make of this? Justin
Moz Bar | | GrouchyKids0 -
Is there anyway to track on-page grader grades for a site?
As opposed to going through the process of inputting the urls and keywords every couple of weeks, is there a way of Moz to send the scores for those keywords and the domain to which the url belongs?
Moz Bar | | DigitalGuru0 -
How Do I Troubleshoot 804 HTTPS Crawl Error?
In my Moz crawl report I get: Crawl Error
Moz Bar | | digium
Moz encountered an error on one or more pages on your site
Error Code 804: HTTPS (SSL) Error Encountered The Moz Help Section only says: 804 HTTPS (SSL) error 804 errors result from a site with misconfigured SSL software. If Moz's crawlers cannot correctly interpret an SSL response for a home page, the crawl ends immediately. My site is publicly accessible on https - https://www.respoke.io/ And I'm not seeing any issues with my certificate. Can anyone help me out? What steps can I take to troubleshoot this error? If SSL is misconfigured, how do I configure it properly?0 -
Canonicals in crawling reports
The crawling reports gives info about several meta data missing, what about the lack of a canonical tag? This would be nice too... and images without alt tag (or empty).
Moz Bar | | KBC0 -
Crawl Diagnostics: Exlude known errors and others that have been detected by mistake? New moz analytics feature?
I'm curious if the new moz analytics will have the feature (filter) to exclude known errors from the crwal diagnostics. For example, the attached screenshot shows the URL as 404 Error, but it works fine: http://en.steag.com.br/references/owners-engineering-services-gas-treatment-ogx.php To maintain a better overview which errors can't be solved (so I just would like to mark them as "don't take this URL into account...") I will not try to fix them again next time. On the other hand I have hundreds of errors generated by forums or by the cms that I can not resolve on my own. Also these kind of crawl errors I would like to filter away and categorize like "errors to see later with a specialist". Will this come with the new moz analytics? Anyway is there a list that shows which new features will still be implemented? knPGBZA.png?1
Moz Bar | | inlinear0 -
Mozscape Update-Why every site is down?
Every site,every competitors site have been down.What type of data Moz used this time to update the index. If the variables are changed can I have the priority variables?
Moz Bar | | csfarnsworth1