SEO Web Crawler IP addresses
-
What are the IP addresses for the SEO Web Crawler?
There is a firewall on my clients website before it goes live, I would like to crawl the site before it goes live, but need to provide the web crawlers IP addreses.
Thank you for your time
-
Hi Jason,
This is Megan from the SEOmoz Help Team. Unfortunately, since we crawl from the cloud, RogerBot's IP address is changing all of the time so we cannot specify a particular IP address. Also, we do need the site to be live before we can crawl it, but you could remove the firewall and block all other bots except rogerbot so it won't get indexed anywhere else before it's ready.
I hope this helps! Please let me know if you have any other questions.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Someone has purchased super similar domain to mine will this effect my site seo overall?
I have a domain name allready registered www.untamedelopement.co.uk and dot com. Someone despite knowing I had this name has registered untamedelopements.com (with the s) is that going to mess up my seo. I'm UK and will be targeting as my keyword Scottish Wedding Photographer and they are Canada? Thanks in advance
Moz Pro | | hemeravisuals130 -
Help talking to an aggressive web team (1st time in the forums)
Hello All! I am a long-time Moz user, but not super technical myself. I'm used to asking a web team to fix something, based on the reports, and they do it. I'm trying to communicate with a really aggressive web team about a site that's showing about 16,000 errors every week on my Moz reports. What I'm trying to figure out is if they are right or not, so I can either ignore them or push harder. Can someone help me with how to explain these to them or let me know if they are right? These are the two biggies I'm struggling with: b. Duplicate page content
Moz Pro | | Renia
Example: https://www.whiteheadindustrial.com/c-104-abrasives.aspx?pagenum=1 and https://www.whiteheadindustrial.com/c-104-abrasives.aspx Their explanation: I wonder if MOZ is not reading the infinite scrolling pages the way that Google does. We have followed Google’s directions for how to let Google know that each page is a successor of the previous one, so no duplicate content. But probably MOZ tool does not view the code on the page the same way. So I think this is not relevant for Google indexint. Second:
Example 2: https://www.whiteheadindustrial.com/p-19708-4-point-mandrel-and-hex-nut-116-hole-18-shank-nocra-13440-m.aspx
https://www.whiteheadindustrial.com/p-30356-working-peanut-knife-no-046-6220-ss.aspx
In Example 2, the only duplicate content I can find is the “Stocking data Current as of… Please note:..” and the Customer Reviews area. c. Duplicate page title
I wonder if MOZ is not reading the infinite scrolling pages the way that Google does; because the duplicate page titles are related to category page I would be super grateful for any guidance!1 -
Hiring for SEO
so sorry if i offend anyone here ...but dang SEO Experts are the used car sale guy of 1999! we have a small eCommerce company in Jacksonville, FL - and we are always looking for improve our SEO - we have been members of moz for almost 10 years now! We have learned alot with all the tools they offer, We have tried at least 5 local and national SEO consultants, agencies and consultants - NONE HAVE WORKED. The time has come for us to hire an in house SEO do you have any advise on how a small business that wants to grow our SEO can weed thru the thousands of SEO "GURU's" out there today - Everyone is an expert. On a side not if you want to move to Florida and help us grow - www.HolmesStamp.com/team
Moz Pro | | BryanCroft1 -
Fresh Web Explorer explorer doesnt seem to work properly
How do I get Fresh Web Explorer explorer to work properly. The feedback I’m getting shows that my brand, Health 2000 had over 4000 mentions on Monday the 13th of January. However when I check out the links that supposedly mention our brand, they bring up general terms related to Health and make no reference to our brand. How can I refine this?
Moz Pro | | Hardley1110 -
How do I see all links to my site in Web Explore?
It appears that the Web Explore looks for exact matches to a page. I'd like to see all matches to any page. Will regular expressions work?
Moz Pro | | Leverage_Marketing0 -
Why are we no longer ranking #1 for our brand on the SEOMoz web app?
Been noticing these anomalies for some time now. I'm from eVenues.com and we've found the rank tracker extremely useful to see how our campaigns are doing. I noticed however that we no longer show up in the app as ranking for our brand? I know that results differ according to location and a myriad other variables but I think we're pretty strong as far as our brand name is concerned. I did a quick double check in incognito and, yep, we're still ranking... Any ideas on this?
Moz Pro | | eVenuesSEO0 -
Why is the SEOmoz crawler crawling the old version of our website?
Hello, I'm a new SEOmoz member. On Dec. 2nd, after completely redesigning our website, we migrated to a new hosting company by switching our DNS to the new server. The vast majority of the URLs have changed and we configured redirects of the old URLs to the new ones. Although, this task is not completed yet. After the migration, I created an account on SEOmoz to be able to track our progress and find the issues to fix to optimize our SEO. For some reason, in the SEOmoz reports it is the old URLs that show up. Unless the crawler does not actually crawl the pages and only uses the indexed pages to generate its report, I don't understand how could this possible. Anyone has a clue? When will the new URLs be indexed by SEOmoz and the major search engines? Thanks for your help!
Moz Pro | | Gestisoft-Qc0