Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz bot not discovering important links (high DA sites link)
Moz bot is unable to crawl and discover my links on the high authority websites like microsoft, linkedin, pinterest, etc. Where is the problem?
Link Building | | TechG0 -
Question about reciprocal link building. I'm not an SEO professional, just a local service business owner.
I did a link page on my website 13 years ago and never took it down. Should we scratch that page all together? Is it ok with Google to do a page on Recommended local service providers. Maybe I can keep some of those reciprocal links if that's the case...
Link Building | | FVLMS0 -
What's Your #1 High Authority Backlink Strategy For 2015?
Hey everyone, I am gathering responses from the SEO community regarding the two questions below. I will feature your (legitmate) responses as part of a study for current SEO backlink methods being used by the SEO community. There are no wrong answers. I look forward to seeing your responses and make sure to add your social profile details. 1. What is your number 1 method of acquiring high authority backlinks?
Link Building | | kirkbowlen
2. What platforms/ software do you use or recommend as part of your method? Thanks, Kirk Bowlen0 -
Starting a new site's link campaign... how to approach it?
Over the last few years I have been building content in my niche that I believe rivals some of the best content out there and deserves some attention. Although I have a plan to produce alot more content which I believe will take the quality and quantity of my content into a position among the top 5 or top 10 sites in my niche in the next 1-2 years, I decided that making that massive investment in content production irrespective of a consistent marketing plan is a recipe for failure because I need the positive feedback loop from site visitors to begin now, not in 2 years. Right now I'm in a position where I'm producing content that I think is better than alot of what's out there, and it's just not ranking the way I believe it should. I think I need to do a legitimate link building campaign to establish the website a little more firmly and put it on more level ground with some of its competitors. In Majestic SEO's "fresh index", most of my site's immediate competition have no more than 500 new domains in their links, though the biggest one has some 2,000. How can any link building effort I might take on possibly compare to links of this scale? Is there some "rule of thumb" for how many quality links I should aim for to get on square ground with some of the competitors on the lowest rungs? And if I try to build that many links at once, do I risk sending signals of untrustworthiness? (Assume I'm not going to be looking for any shoddy links, and in general will aim to follow Google guidelines.)
Link Building | | guitarsites0 -
Why is OSE not showing Google+ bio link in backlink profile?
Hi fellow mozzers I wonder why opensiteexplorer (ose) isn't showing Google+ bio links in backlink profile reports of websites that get a link from the "about" page of a Google+ account (+ contributor link). Thanks for sharing your thoughts. Jacob
Link Building | | Jacobe0 -
Back linking to t foreign sites
One of our major competitors seems to be linking to an Asian speaking website that has a blog/product review formant. I was wondering how they are achieving this. Are there any non English /Asian sites worth submitting to and what is the best way to go about this if English is the only language you speak! And of course is it worth while doing this from an SEO perspective.
Link Building | | Hardley10 -
I'm thinking about buying a competitor and 301 redirecting? How much SEO value?
I'm thinking about buying a competitor and 301 redirecting their site to mine. The high level stats are as follows. My site has a DA of 46 and the homepage has a PA of 55. 375 root domains, 170,000 links. The site sells millions of dollars worth of product each year. The competitor (who I had never heard of) has a DA of 58 and a homepage PA of 64. They have 634 root domains and 260,000 links. They aren't selling much of anything (less than 100,000 per year). We might be able to operate their site but I'm concerned about maintaining 2 platforms. My question is about the value of buying this site and 301 redirecting it to my site. Would this create long term SEO value or not? Any examples that have been documented are greatly appreciated.
Link Building | | bradwayland0 -
What's the typical response time to link building email requests?
Hello Forum, We're about to embark on a link building campaign and were curious about how long, on average, it takes to get a response to an email requesting links to our page. We're trying to come up with a timeline estimate for our campaign. Thanks
Link Building | | pano0