Moz can't crawl domain due to IP Geo redirect loop
-
Hi,
I'm trying to crawl our domain www.salvationarmy.org.au via my Moz account and it only ever returns results for one page when it should be crawling more than 3,000 pages.
In talking to support, they have said that because of the redirect we have in place it is creating a 302 loop and therefore not delivering results. Usually in this case I would obtain Moz's IP addresses and add them to the redirect settings as an exception, but Moz have said they use cloud-based services for crawling so the IPs change all the time.
Does anyone have any idea how to solve this issue? At this point I've paid for a year's subscription to a product I can't use.
Thanks,
Mel
-
Hi,
If you have manually set up your geo redirect in your htaccess then you could modify your rules to redirect only if not Moz’s crawler (rogerbot) like this:
uk redirect
RewriteCond %{HTTP_USER_AGENT} !=rogerbot
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^GB$
RewriteRule ^(.*)$ http://uk.abcd.com$1 [L]
Which means both conditions must be satisfied before the redirect happens, the user agent must not be rogerbot, and then it checks the country code. You may have to adjust it a bit depending on your setup but it’s just the same as adding an exception based on IP, so if you could already do that you can set up a user agent condition just as easily.
If you’re using PHP you could use $_SERVER['HTTP_USER_AGENT'] wrap your geoip function with something like:
if($_SERVER['HTTP_USER_AGENT'] != 'rogerbot' ){
You will have to check if it is not empty before you implement it (or work it into your code) as some servers have $_SERVER['HTTP_USER_AGENT'] as not set.
Thanks, hope that gives you a few ideas to try!
Tom
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does MOZ have a Flash test tool?
I want to test my websites and see if they use Flash, is there a flash check tool like on SEO tool kit here on MOZ? Thanks, Lance
Link Explorer | | BlueprintMM0 -
How can I tell what pages have links TO this page?
Hello, Mozzers! My client has a rather messy situation, with redirects for HTTP vs. HTTPS, and two different home pages, both in subdirectories. I could explain the whole setup, but I don't think it's relevant for my question, which is this: OSE is showing three links to the page in question, but doesn't tell me WHAT those links are. Am I just missing the information somewhere? Please see screenshot for details. Thanks so much! ~ Scott view?usp=sharing
Link Explorer | | measurableROI0 -
Hey greetings I have one query as If i want to know what type of keywords are my competitors are using then how can i analysed them and make sure that the keywords are targeted properly.Kindly reply asap
Hey greetings I have one query as If i want to know what type of keywords are my competitors are using then how can i analysed them and make sure that the keywords are targeted properly.Kindly reply asap
Link Explorer | | vikiran0 -
Why Moz Doesn't See or Count Our Backlinks?
Hi Moz Community! We have been working hard to improve our Moz metrics, measuring against a high ranking competitor to help us set our goals. Our Majestic and Webmaster Tools find tens of thousands of external backlinks pointing to our domain. That's all well and good. Moz's Open Site Explorer, however, only finds 900 total links - including internal links! This being the case, we have worked diligently to build a variety of great external backlinks, creating Bitly links and encouraging clicks on those through social promotion. Yet, our competitor has over 7,000 external backlinks in Moz's index, while ours is not growing relative to their number of backlinks. Can anyone share with us what they do to tell Moz about their backlinks? We already know we have many more backlinks than our competitor, from trusted domains with good authority, yet it seems Moz is not discovering them. We just want to understand how to use these Moz metrics to create meaningful calls-to-action. Otherwise, it seems like a gargantuan waste of time, and our team has difficulty getting buy-in from our company to put time and assets toward tasks based on our Moz numbers!
Link Explorer | | RegistrarCorp0 -
How Does Moz Assign Domain Authority?
When I use Site Explorer to test the following URL, http://jennathuening.results.net/site/index.php, it renders a domain authority of 47. (Clients site on real estate host) There are no page social metrics: no Facebook share no Facebook likes no Twitter tweets no Google+ plus 1's no "just discovered" in 60 days no one has made a post in 3 weeks much or most of the content is duplicate content across all affiliate sites inbound links - only one link - a 301 redirect http://jennathuening.results.net/ Images - 18 (Alt tags missing: 18) Created - 1998-10-15 Registered Org - REMAX RESULTS is associated with ~416 other domains Whois SEO Score 68% AND YET IT HAS A ROOT DOMAIN OF "47" and page authority of 33 More confusing, its page authority of 33 is at that top on blue background and then page 30 just listed below. When I use Site Explorer to test the following URL, http://www.homedestination.com, it renders a domain authority of 26. (Site personally owned and continually update with highly optimized fresh, relevant, quality content with grade "A" moz ranked pages and blog posts) 92 Facebook share 68 Facebook likes 13 Twitter tweets 420 Google+ plus 1's 2 "just discovered" in 60 days Images 25 (Alt tags missing: 0) inbound links - on 48 Root Domains Created - 2002-08-15 9 new indexed blog post so far in August / 19 new indexed blog post in July Whois SEO Score 100% AND YET IT HAS A ROOT DOMAIN OF "26" and page authority of 36 While I am sure there are many other factors as well; it still is hard to see where a "smaller guy" could ever be given an even opportunity to play in the same SEO arena. Could you help me understand better? I like to build a stronger marketing and optimization strategy as we are just beginning AdWord campaigns. | SEO Score | 100% |
Link Explorer | | jessential1 -
Does the inbound links report include links to all pages of the domain being researched?
If I enter 'abc.com' am I only getting results for 'abc.com' or will I get results for the internal pages of 'abc.com' as well (i.e. 'abc.com/page1.html)? There is a bit of a discrepancy between these results and inbound link results in semrush for example. Then again it seems whenever you use different tools to measure the same thing you get wildly varying results. How do you all deal with that?
Link Explorer | | AISEO0 -
MOZ Tool Bar Query
Hi there, I have a quick question about how pages are ranked on the MOZ tool bar. If you view this page (with the MOZ tool bar showing on your browzer), it has strong metrics: http://www.bristol.gov.uk/page/leisure-and-culture/bristol-collection-venue-hire-0 However, when you click the 'Weddings' link a go one page further into the site, is has no metrics: http://www.bristol.gov.uk/page/leisure-and-culture/wedding-venue-hire Is this because the page hasn't been crawled? Getting a link here is still good?
Link Explorer | | jennie.evans0 -
OSE Stats - Number of Unique C IP's and PA/DA keep going up and down. Why?
For the past 3 months (if not more) I have been checking my sites stats on OSE and have noticed drastic decreases and increases in unique C-IP's, ranging from 600 to 1200. I have also noticed the PA & DA going up and down by 5 points. We have a talented team that work hard to market our business ethically with strong content and PR relationships, white-hat link building all the way, active social media and a dedicated on-site technical team. The data from MajesticSEO seems to be a lot more steady and consistent. Why is OSE showing drastic changes in 1 to 2 week intervals? Many thanks Ross
Link Explorer | | David_Connor0