How to turn off automated site crawls
-
Hi there,
Is there a way to turn off the automated site crawl feature for an individual campaign?
Thanks
-
No problem!
Thanks for the suggestion!
I will pass that on as feedback
Eli
-
Hi Eli,
Thanks for your response.
I'd really suggest that the functionality to prevent the site crawl feature is built into the campaign tool. It would take dev effort to disalow the Moz bot where I'm based.
Thanks
-
Hey!
Thanks for reaching out to us!
Moz uses a crawler called Rogerbot to crawl your site and populate the "Site Crawl" section of your Campaign. Roger isn't a search engine index, so if URLs aren't blocked by the robots.txt file, then Roger will crawl them!
If you'd like to remove pages from the crawl, you can disallow them in the robots.txt file. Here's what that exclusion would look like if you were wanting to keep him from crawling any pages on your site:
User-agent: rogerbot
Disallow: *I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
In Moz Campaigns, how are competitor domains tracked if they redirect their site?
Hello! One of our competitors (Company A) that we've tracked in Moz for a long time recently merged with another company (Company B) and redirected their whole site to Company B's site. Will our competitor tracking still work as-is? Or do we need to make an adjustment? I'm reluctant to delete Company A from our competitor tracking, because we will lose all of that data. But if all of the keywords are slowly going to drop off as Google starts showing Company B results only, it may be the only option. Any help is appreciated! Thanks!
Moz Bar | | PrimeFoodTeam0 -
Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt
Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this: User-Agent: rogerbot Crawl-Delay: 5 Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop. Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks
Moz Bar | | AllAboutShapewear2 -
Moz crawler only crawls one page?!
Hello there, I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled. We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working. Hopefully someone can point us at the right direction. Thanks in advance, Jeremy
Moz Bar | | mediaxplain.nl0 -
The Moz Spam Score tells me my site has too few backlinks for such a large site. How many links per page would I need to not trigger this filter and stop appearing spammy?
Hello! One of my sites is triggering the 'too few backlinks for large site' filter. I am wondering how many backlinks I need so as not to trigger this. Many thanks for your help. Toby
Moz Bar | | T0BY0 -
I'm checking keyword difficulty for two different sites. Would love to view the results by site instead of just one large list. Is that possible? Or would it just be easier to keep the lists separate in Excel and just import when I want an updated report?
I have keyword lists for two sites. Is there a way to label them in the keyword difficulty tool (List A, List B) so I can just view results for a particular site? Or do I need to run the report with List A, export results, delete those keywords, then run the report for List B?
Moz Bar | | JohnNovakLV0 -
Internal Links Count in Crawl Report
My understanding of the 'Internal Links' results in a moz crawl report is that it represents the number of links on the given page that link to other pages on the same site.Assuming this is a correct assumption: We recently ran a crawl report on www.phase1tech.com. Some of the pages are coming back with a large amount of 'internal links'. These 2 pages for example are showing 800 internal links: http://www.phase1tech.com/Upcoming-Events
Moz Bar | | AISEO
http://www.phase1tech.com/Contact Then there are a number of pages coming back with 705 Internal Links, including: http://www.phase1tech.com/Dalsa-CameraLink-Cameras
http://www.phase1tech.com/Hitachi-CameraLink-Cameras At best there are approximately 70-80 links on these pages. Where are these large counts coming from? Is there a means to see what the links being reported on are? At the same time the 'Too Many On-Page Links' indicates 'No' for some pages with a high number of links, and 'Yes' for pages with a low number of links. For example: http://www.phase1tech.com/Baumer-SX-Series
Too Many On-Page Links: Yes
Internal Links: 2
What's up with that?0 -
Site Wide Content Analysis
Are there tools that can effectively analyze your content site wide and provide recommendations?
Moz Bar | | SpadaMan0