SEO Web Crawler IP addresses
-
What are the IP addresses for the SEO Web Crawler?
There is a firewall on my clients website before it goes live, I would like to crawl the site before it goes live, but need to provide the web crawlers IP addreses.
Thank you for your time
-
Hi Jason,
This is Megan from the SEOmoz Help Team. Unfortunately, since we crawl from the cloud, RogerBot's IP address is changing all of the time so we cannot specify a particular IP address. Also, we do need the site to be live before we can crawl it, but you could remove the firewall and block all other bots except rogerbot so it won't get indexed anywhere else before it's ready.
I hope this helps! Please let me know if you have any other questions.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Need Moz SEO Wordpress Plugin With API
Re: Moz WordPress Plugin? Hi guys,
Moz Pro | | mrezair
I need some Moz SEO Wordpress Plugins For my website working with Moz API. I've already found Moz DA-PA Checker plugin Moz DA-PA Checker But Need SEO Plugins too. Any Suggestion will be appreciated.0 -
Redirects, SEO and More
Website errors remittent of Chernobyl. First of all, I ask a lot of questions of the community and I don't do a lot of answering. For that, I am sorry! I own a small SEO company and the majority of my clients are upstarts just like myself. We try to do the best we can on a very limited budget.
Moz Pro | | underdogmike
With that said, the client website in question, http://bestdefensega.com has been a struggle to rank in the metro atlanta lawyer market.
Here's the story. My client is a new solo attorney that practices criminal law and personal injury law. We launched bestdensega.com to focus on criminal defense and my client has also paid AVVO for a website as well. We recently decided to make a monster website that would have 50+ pages and be representative of all that he practices.
I recently put a temporary redirect on bestdefensega.com and pointed it to the avvo website while I was rebuilding the website. I did this using a plugin.
This morning I decided that I wanted to remove the redirect, so I deactivated and deleted the plugin. The redirect is still in effect. I suspect that this may be a cacheing scenario.
How do I undo a redirect?
Also, I noticed that the AVVO site has all of a sudden shot up to the top of the search results and bestdefnesega.com has dropped past the 3rd page for our keywords. Have I lost all of bestdefensega.com's link juice? Also, bestdefensega.com has a spam score of 7. Why? There are no shady links and everything I do is on the up and up. Also Open Site Explorer says that bestdefensega.com doesn't have any inbound links.
Google WebMaster Tools says that the site has of 500 inbound links.. Frustrated, Thanks0 -
Found hidden pages of outbound links created via ex-SEO consultant. Best way to detect other possible problems?
We paid for an SEO contract in addition to our new website design (same company did both) and after 12 months cancelled the SEO. I have been very suspicious ever since of our bad page rank and general lack of traffic (despite my efforts) and today found a hidden page of outbound links. Currently in shock that this happened although my own fault for lack of due diligence. The SEO consultants were very unhappy that I cancelled the contract so I am worried about the extent of bad links or negative google juice they may have created (god knows what else they may have done). So my questions are: How can I detect any other (potentially hidden) problems? How can I recover from this - any right/wrong way to approach google? What is the best way to bring this up with the SEO consultants? Thank you in advance.
Moz Pro | | marketing-gal0 -
The pages that add robots as noindex will Crawl and marked as duplicate page content on seo moz ?
When we marked a page as noindex with robots like {<meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">noindex</a>" />} will crawl and marked as duplicate page content(Its already a duplicate page content within the site. ie, Two links pointing to the same page).So we are mentioning both the links no need to index on SE.But after we made this and crawl reports have no change like it tooks the duplicate with noindex marked pages too. Please help to solve this problem.
Moz Pro | | trixmediainc0 -
Who else has inaccurate reporting? SEO Moz BUG
I have been in contact with support to little avail no timelines for the correction kind of leaves me out in the cold. I have loved SEOmoz in the past but this recent bug is costing me time and money. Can someone please tell me how they are working around all these issues in rank reporting and a zero accuracy rate on page report cards? bugs are holding up my SEO efforts. SEOmoz has been in touch but has provided no timelines. My programs are suffering. I found inaccuracy in each report metric and now have very little faith that this is going to be corrected in a timely fashion. I rely greatly on the wonderful tools SEOmoz provides and now I am flying blind. Who else has been hit by the bug? And, what has support done for you to help?
Moz Pro | | ericajane0 -
SEO Tools; which ones can you NOT live without?
There's loads of free and paid SEO tools out there and I'm trying to find out which ones make up a common denominator for most professional SEOs.
Moz Pro | | MassivePrime0 -
SEO Web Crawler - Referrer Lists XML Sitemap URL
Hello!, I recently ran the crawl tool on a client site. Opening up the file, I noticed that the referring URLs listed are my XML sitemaps and not (X)HTML pages. Any reason or thoughts behind why this is happening? Thanks!
Moz Pro | | MorpheusMedia0 -
SEO Targeting: Ideal monthly keyword search volume?
What's your SEO sweet spot for monthly keyword search volume*? At what volume do you actually expect to see traffic? (I know this depends heavily on competition for said keywords). (*Currently measuring volume via Google AdWords Keyword Tool / SEOmoz keyword difficulty tool. || Comparing exact match and broad match.)
Moz Pro | | JessicaCox0