Unsolved Site Crawler not working but on-demand crawler working
-
Hi,
In Moz pro, when using Site crawler (or recrawl), we are seeing message site is banned. But when using on-demand crawler, it could generate report successfully.
I just like to know if in both these cases, it is roberbot that is used!
And kindly note, site crawler was perfectly working before. So the required setup is already in place from long time. Site crawler ban issue started appearing from nov/dec 2023. .
Could you please us understand how could we possibly make site-crawler work?
I am happy to provide more details if you need any.Thanks
-
Hi,
This question requires help from MozPro.
Site Crawler is not working because it is missing request header 'user-agent' when we investigated the logs in our system and it got banned because of this reason.
On-demand crawler is still working because it has request header 'user-agent' and our system approved it hence able to generate report.Could you please look into this issue of no-user-agent request header?
Your response is much appreciated.Thanks
-
Hi,
I will double-check with firewall settings in our servers. Could you please share moz-pro site-crawler roger bot IP addresses/range? We will verify against our firewall rules.
Thanks
Shashi -
I am looking for roger bot site crawler IP addresses Please provide.
Thanks
-
@Aditi_08
Could you please help me on how to get IP addresses of Site Crawler? Just please note, Site Crawler is working before November so IP addresses were not blocked.Like it is mentioned before,
- no change in robots.txt
- no issue with rate limiting
- no changes in site-crawler configuration
-
@gilesd If you're experiencing issues with Moz Pro's Site Crawler showing that the site is banned while the On-Demand Crawler works fine, it might be due to changes or updates since November/December 2023. Both tools likely use the same crawler, "rogerbot," but differ in their operational schedules. The problem could be due to rate limiting or blocking by your server, IP blocking, changes in your robots.txt file, or updates in the Site Crawler configuration. To resolve this, check your robots.txt file to ensure it allows Moz's crawler, review server logs and firewall settings to ensure the crawler’s IP addresses aren’t blocked, and adjust rate limiting settings if necessary. Also, double-check the settings in Moz Pro to make sure there are no configurations causing the issue. If the problem persists, contact Moz support with detailed information about the error messages and any recent changes to your site’s configuration. Regular monitoring of your site’s interactions with automated tools and coordinating with your hosting provider can help prevent such issues in the future.
-
I am not sure why my reply not appearing here. Just for confirmation, replying again,
I like to confirm you -
There is no modification in Robots.txt
No issues with rate limit
Moz Pro settings are not changedWe are looking for your help to identify the issue.
Thanks
-
Thanks for your trouble shooting tips.
I assure you there has been nothing changed in robots.txt file or any settings in MozPro.
And there is frequency limit, Site Crawler triggers only once in 2 weeks.Thanks
-
Hi, gilesd
In Moz Pro, when using the Site Crawler or Recrawl, we also received a message indicating the site was banned. However, the on-demand crawler could generate the report successfully.
To address your question:
Robots.txt Configuration: Both the Site Crawler and on-demand crawler should be using the same robots.txt file unless there's been a recent change. Ensure your robots.txt hasn't been updated to block specific user agents.
IP Blocking or Rate Limiting: Some web servers or security settings might block or limit access based on IP or request frequency. The Site Crawler might be hitting these limits, whereas the on-demand crawler, being less frequent, avoids these blocks.
Moz Pro Settings: Double-check the Moz Pro settings to see if there have been any changes or updates to how the Site Crawler operates compared to the on-demand crawler. Any recent updates might have altered how the Site Crawler interacts with your site.
Thanks,
Hamza Zubair -
Hi, gilesd
In Moz Pro, when using the Site Crawler or Recrawl, we also received a message indicating the site was banned. However, the on-demand crawler could generate the report successfully.
To address your question:
Robots.txt Configuration: Both the Site Crawler and on-demand crawler should be using the same robots.txt file unless there's been a recent change. Ensure your robots.txt hasn't been updated to block specific user agents.
IP Blocking or Rate Limiting: Some web servers or security settings might block or limit access based on IP or request frequency. The Site Crawler might be hitting these limits, whereas the on-demand crawler, being less frequent, avoids these blocks.
Moz Pro Settings: Double-check the Moz Pro settings to see if there have been any changes or updates to how the Site Crawler operates compared to the on-demand crawler. Any recent updates might have altered how the Site Crawler interacts with your site.
Thanks,
Hamza Zubair
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Duplicate Content
We have multiple collections being flagged as duplicate content - but I can't find where these duplications are coming from? The duplicate content has no introductory text, and no meta description. Please see examples:- This is the correct collection page:-
Technical SEO | | Caroline_Ardmoor
https://www.ardmoor.co.uk/collections/deerhunter This is the incorrect collection page:-
https://www.ardmoor.co.uk/collections/vendors How do I stop this incorrect page from showing?0 -
Unsolved Bug in site crawl analysis - 308 redirect flagged as temporary
Hi, we have some 308 redirects on our website, which are permanent redirects, but the site crawler is flagging them as temporary currently. Screenshot 2022-02-10 14.24.26.png
Moz Pro | | pm-mbc0 -
Solved Site Crawl Won't Complete
How can I start/restart a new site crawl? I requested one 2 days ago on one of my sites, and it won't complete. It's only 150 pages -
Product Support | | PaulBarrs0 -
Solved Why is MOZ crawl taking so long?
I began my site crawl on November 3rd and now it is November 7th and it is still "in progress". Why is this happening?
Product Support | | CarisaS_Wenda0 -
Automatically Check List of Sites For Links To Specific Domain
Hi all, Can anyone recommend a tool that will allow me to put in a list of about 200 domains that are then checked for a link back to a specific domain? I know I can do various link searches and use Google site: command on a site by site basis, but it would be much quicker if there was a tool that could take the list of domains I am expecting a link on and then find if that link exists and if so on what page etc. Hope this makes sense otherwise I have to spend a day doing it by hand - not fun! Thanks,
Moz Pro | | MrFrisbee
charles.0 -
Site Explorer shows links as followable but they have nofollow tags
Hello, I am looking at site explorer and sites linking to my site moneyfact.co.uk. I've got thousands of links showing as 'followable' but when i check them they have rel="nofollow" tags. e.g: http://www.dianomioffers.co.uk/partner/moneyfacts.co.uk/brochures.epl?partner=93&partner_id=93&partner_variant_id=33 Why would they show as followable when the links are nofollowed? Thanks Steve
Moz Pro | | SteveBrumpton0 -
Looking to Hire an SEO Expert for Travel Site
Hello Everyone, I hope this is OK to post here. I run a travel website and we operate tours abroad. We are slowly expanding after nearly 10 years in the field. I started the company from scratch and currently do my own website design and SEO work. However, over the last couple of years I've lost the SEO connection due to being so darn busy. I started account here with SEO Moz in order to get the ball rolling again. We continually rank high for keywords, publish superb content by great writers, and overall are doing fine. I would just like to ensure this keeps happening and that we have an expert on board to help us in sustainable organic SEO practices. What I'd like to do is hire an SEO expert to help with our site. It might be part time (10 or so hours per week, perhaps?) Anyhow, I would like if this person used SeoMoz and really had their fingers on the pulse of the SEO world. I want all white-hat SEO, help with page optimization, internal link structure, and help getting quality external links. I know that over optimization is something that is now on the forefront of SEO work, so of course keeping this in mind helps. I would like to be kept up to date (perhaps weekly overviews of data and how we are doing) and am willing to make changes as quickly as needed on the site. I would like to see how we rank with competitors and what we need to do to stay on top - for the long term (no short sighted decisions). I want to learn as we progress, not because I want to do the SEO work myself, I just enjoy knowing what is going on. I will not micromanage. I am too busy for that : ) I am not sure what amount SEO experts are paid, but I have a slightly flexible budget. I really want someone will communicate when needed and is thorough in their responses. Attention to detail is a must. I would prefer an SEO expert located in the US but who also knows how to access / think about markets abroad (Canada/UK/Australia). I suppose we can discuss pay once I have a shortlisted group. I am looking to hire and work with someone starting pretty much immediately (even in May). For starters, I suppose our budget might be $500 to $1500 per month. For the first month or so, we might offer payment on a stipend basis just so we can ensure whoever is hired is the perfect fit. It would be great if this person had worked on travel sites before or has at least traveled abroad a few times. But, no worries either way. Thanks so much! Thomas at [email protected]
Moz Pro | | journeybeyondtravel0 -
Will Open Site Explorer ever show number of links over time?
OSE is an amazing tool, but do you guys at SEOmoz have any plans to develop it so we can track numbers of links over time. I need to demonstrate to clients how the link building is going, and this would be a great quick report to see how many links you found on a given day, month, year, etc. A bit like magesticSEO backlink history graph, but better 🙂
Moz Pro | | timwills0