SEO Web Crawler IP addresses
-
What are the IP addresses for the SEO Web Crawler?
There is a firewall on my clients website before it goes live, I would like to crawl the site before it goes live, but need to provide the web crawlers IP addreses.
Thank you for your time
-
Hi Jason,
This is Megan from the SEOmoz Help Team. Unfortunately, since we crawl from the cloud, RogerBot's IP address is changing all of the time so we cannot specify a particular IP address. Also, we do need the site to be live before we can crawl it, but you could remove the firewall and block all other bots except rogerbot so it won't get indexed anywhere else before it's ready.
I hope this helps! Please let me know if you have any other questions.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO impact of redirecting high ranking mirror site to the main website
During SEO audit for a client I noticed that they had over a dozen duplicate websites that are carbon copies of the main website. This was done via CMS platform and DNS. One of the mirror sites has about 400 indexed pages and has Moz DA of 42 and 137k External Equity-Passing Links. Full metrics comparison is attached. I originally planned on doing rel="canonical" on the mirror site but the CMS vendor never even heard of it and is refusing to implement it in the header. My only other option is doing one to one 301 redirects. Since the mirror site ranks well, even competes with main domain for some positions on the 1st page of SERP, what will be the impact after the redirects? Is doing 301's still the best option? Thanks! PrUpN3q
Moz Pro | | dasickle0 -
Angular.js + Crawlers
I am working with a site that recently deployed Angular.js on the site. From an SEO standpoint its a little more tricky than we thought. We have deployed a couple updates to render pages for the bots but we not seeing changes in Moz weekly reports. When it comes to Angular.js, will the Moz bots read/access the site the same as the other major engines? I'm trying to figure out if our deployments are working or if there's something off in the Moz reports. Thanks.
Moz Pro | | JoshKimber0 -
SEOmoz crawler not crawling my site
We set up a new campaign in SEOmoz on Friday. It is my understanding that the preliminary crawl can cover up to 250 and this has been our experience in the past. However, the preliminary crawl only went through 2 pages. This is a larger eCommerce site with many pages. Any ideas why more pages weren't crawled? We set up the campaign to track at the root domain level.
Moz Pro | | IMM0 -
Can not start a SEO MOZ campaign?
I keep getting this message, I have a 301 from my domain to my domain with the www prefix. What's the deal? We have detected that the domain www.photosbykristopher.com and the domain photosbykristopher.com both respond to web requests and do not redirect. Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here.
Moz Pro | | KrisPhoto0 -
One Follow Link from each subdomain/domain best for SEO?
Hi there, we have a lot of customers linking to our homepage from a subdomain of our domain or from an external domain. From SEO point of view is it best that they all only have one DoFollow Link and the rest of the links are NoFollow? Or does this have no matter at all? The problem is that we have like 900.000 links but only from round about 1000 different root domains. If we put like half of the links on NoFollow our ratio of links/ different root domains gets better. Thanks for answering! Sebastian
Moz Pro | | Sebastian230 -
Why cant I hook up facebook to SEO Moz Campaign Tool?
Greetings from the digital epicentre otherwise known as Wetherby Uk 😉 I want to track sentiments ie mentions, slurs fliratations of a brand on Facebook. So to start out with I thought 'Id hook up SEO moz social tracking service as illustrated here: http://i216.photobucket.com/albums/cc53/zymurgy_bucket/cant-add-facebook-url.jpg But whilst i could add my twitter account I caount not add my favebook page:http://www.facebook.com/david.honan.98I added david.honan.98 in the url box and a number of other versions but none successfully hooked up. Am enetering the wrong url or is SEO moz socail media facebook plugin jinxed? Thanks in advance, David
Moz Pro | | Nightwing0 -
Conflicting Data with Open Site Explorer & SEO Moz Toolbar
Hi there, Why is my SEOmoz toolbar showing me higher linking root domains than open site explorer? For example, for this site: http://thisisouryear.com/, the linking root domains in SEOMOZ show 1,015 and in open site explorer it says it has 35. I have noticed this for other sites as well. Which one is right? And why are they different?
Moz Pro | | qlkasdjfw1 -
Incorrect domain authority result on SEO tool bar and OSE
The SEO tool bar is returning what I believe to be an incorrect domain authority of 71 and showing 24,356,141 lins from 153,051 domains. The OSE is also returning 71 as domain authority. Anyone know what could be doing this? Thanks. Jason
Moz Pro | | jayderby0