Unsolved The Moz.com bot is overloading my server
-
. How to solve it?
-
maybe crawl delay will help.
-
@paulavervo
Hi,
We do! The best way to chat with us is via our contact form or direct email. We also have chat within Moz Pro.
Please contact us via [email protected] or https://mza.seotoolninja.com/help/contact
We will be happy to help.
Cheers,
Kerry. -
very nice brother i like it very good keep it up !
-
very nice !
-
does the moz team even monitor this forum?
-
If the Moz.com bot is overloading your server, there are several steps you can take to manage and mitigate the issue effectively. First, you can adjust the crawl rate in your
robots.txt
file by specifying a crawl delay for the Moz bot using directives likeUser-agent: rogerbot
andUser-agent: dotbot
, followed byCrawl-delay: 10
to make the bot wait 10 seconds between requests. If this does not suffice, you can temporarily block the bot by disallowing it in yourrobots.txt
file. Additionally, it's a good idea to contact Moz’s support team to explain the issue, as they may offer solutions to adjust the crawl rate for your site. Implementing server-side rate limiting is another effective strategy. For Apache servers, you can add rules in your.htaccess
file to return a 429 Too Many Requests status code to the Moz bots, while for Nginx servers, you can set up rate limiting in your configuration file to control the number of requests per second from a single user or IP address. Monitoring your server’s performance and log files can help identify specific patterns or peak times, allowing you to fine-tune your settings. Furthermore, using a Content Delivery Network (CDN) can help distribute the load by caching content and serving it from multiple locations, reducing the direct impact on your server caused by crawlers. By taking these steps, you can manage the load from the Moz.com bot and maintain your server’s stability and responsiveness.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved 503 Service Unavailable (temporary?) Rogerbot takes a break
A lot of my Moz duties seem to be setting hundreds of issues to ignore because my site was getting crawled while under maintenance. Why can't Rogerbot take a break after running into a few of these and then try again later? Is there an official code for Temporary Service Unavailability that can smart bots pause crawls so that they are not wasting compute, bandwidth, crawl budget and my time?
Product Support | | awilliams_kingston0 -
520 Error from crawl report with Cloudflare
I am getting a lot of 520 Server Error in crawl reports. I see this is related to Cloudflare. We know 520 is Cloudflare so maybe the Moz team can change this from "unknown" to "Cloudflare 520". Perhaps the Moz team can update the "how to fix" section in the reporting, if they have some possible suggestions on how to avoid seeing these in the report of if there is a real issue that needs to be addressed. At this point I don't know. There must be a solution that Moz can provide like a setting in Cloudflare that will permit the Rogerbot if Cloudflare is blocking it because it does not like its behavior or something. It could be that Rogerbot is crawling my site on a bad day or at a time when we were deploying a massive site change. If I know when my site will be down can I pause Rogerbot? I found this https://developers.cloudflare.com/support/troubleshooting/general-troubleshooting/troubleshooting-crawl-errors/
Technical SEO | | awilliams_kingston0 -
Moz crawler is not able to crawl my website
Hi, i need help regarding Moz Can't Crawl Your Site i also share screenshot that Moz was unable to crawl your site on Mar 26, 2022. Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.
Technical SEO | | JasonTorney
my robts.txt also ok i checked it
Here is my website https://whiskcreative.com.au
just check it please as soon as possibe0 -
Help us with Moz Academy!
Hey Q&Aers! I just wanted to create a discussion thread here to talk solicit some feedback and help with Moz Academy! We want your help coming up with future lessons so we’re getting you the content you need. So, think about what you’d or your team would like to learn from Rand and others at Moz. Let us know! “So, what is Moz Academy?” you may ask. Well, Moz Academy gives subscribers the power to be better marketers, which will enable them to use Moz’s products with more depth and awesomeness. We want to provide a hub of marketing knowledge that will create a stronger community where people can teach each other while using Academy as a frame of reference. The key to this project is empowering you to kick even more ass than you already do! If you have any questions about Moz Academy, feedback about lessons, or ideas on a future curriculum please feel free to get involved in the discussion below. Looking forward to see what everyone comes up with.
Moz Pro | | Nick_Sayers1 -
Moz is officially demigod-approved!
So, I was reading Percy Jackson & the Olympians: The Battle of the Labyrinth tonight, and found a description of a graffiti tag in the Labyrinth that says "MOZ RULZ". What do you think? Is one of the Mozzers a demigod who has explored and survived the Labyrinth? 🙂 (See attachment for book quote.) F1WmT5r
Moz Pro | | AdamThompson4 -
Who else has inaccurate reporting? SEO Moz BUG
I have been in contact with support to little avail no timelines for the correction kind of leaves me out in the cold. I have loved SEOmoz in the past but this recent bug is costing me time and money. Can someone please tell me how they are working around all these issues in rank reporting and a zero accuracy rate on page report cards? bugs are holding up my SEO efforts. SEOmoz has been in touch but has provided no timelines. My programs are suffering. I found inaccuracy in each report metric and now have very little faith that this is going to be corrected in a timely fashion. I rely greatly on the wonderful tools SEOmoz provides and now I am flying blind. Who else has been hit by the bug? And, what has support done for you to help?
Moz Pro | | ericajane0 -
Why does SEOmoz assign such a high Domain Authority to subdomains on blogspot, typepad, wordpress.com, etc.?
I thought it was generally accepted that the search engines treat joesmith.wordpress.com and www.wordpress.com as completely different domains. In which case, a cute but obscure blog like http://monkeysalwayslook.typepad.com/ has no business having a DA of 95.
Moz Pro | | CMC-SD0 -
How can the differences between Moz and Majestic's link totals be explained?
Hello, Newbie here so this might be a really daft question.... We've a few accounts and are trialling different software - across a few campaigns, we're seeing big differences in link numbers from different software (namely Moz and Majestic). I'm assuming there is a really good reason for it - just can't find what it is! Thanks in advance!
Moz Pro | | jghull0