Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Backlink not index on MOZ
Hi All, help me... why backlink for https://gobiz.co.id/pusat-pengetahuan/aplikasi-kasir/ at https://aldhifajar.com/spots-aplikasi-kasir-simple-untuk-setiap-jenis-usaha/ not index at Moz? how it problem on web?
Link Building | | masirwin9180 -
Arrrg . . . Just can't seem to get there
http://www.electricianinperth.com.au is one site that me and my colleagues are constantly struggling with. It has a page rank of 5 which beats a majority of the competitors, but when it comes to Google Australia searches such as Perth Electrician and Electrician Perth etc etc, we just can't seem to get there and the rankings keep fluctuating and dropping. We backlink and update the pages on a regular basis Any ideas? - Could it be the custom Wordpress/CMS system?
Link Building | | lewisjosep70 -
Is too high a frequency of 'money' keywords backlinks considered factor for Penguin penalties, even if the money keywords are on reputable pages ?
Is too high a frequency of 'money' keywords backlinks (eg. a money keyword backlink for moz.com would be "Seo tools") a considered factor for Penguin penalties even if the money keywords are only on reputable pages, with decent PA, DA and trust ?
Link Building | | jpeg800 -
My website Moz Page Authority
My website is Best Comedy Tickets. The current homepage has a Page Authority is 41, mR : 4:53 and Domain Authority is 31. However all my sub pages are basically non existent. All the other pages on the site are a Page Authority of 1, mR : 0.00. Why is this and how do I increase this ? Are there any tips or tricks to increase subpages ? I appreciate all the feed back and am grateful for any tips or helpful advice.
Link Building | | JosephSantiagoNYC0 -
I am switching shopping cart providers, and I cannot keep the same URL's we've had for the past 10+ years.
This applies to our product and category pages. What is the best way to limit the impact of this?
Link Building | | absoauto0 -
Why Breadcrumbs don't work on my web page?
I tried 2 types of breadcrumbs plugins. Last - Yoast
Link Building | | NadiaFL
Breadcrumbs. But result is the same - they don't show up in the footer. I
followed direction and made settings - result is the same. Any ideas? Thank you! http://oasisoftheseasallureoftheseas.com/0 -
Links from Directory's
I have been looking at the Directory's recommended on the SEOMOZ site. All of those that I have looked into do not appear to have a page rank for the actual page that my link would be appearing on. They all appear to offer N/A as the reply.Is this a problem? Thanks in advance for any replies!
Link Building | | Babyshoe0 -
OSE shows links on sites but can't find links
Hi mozzers, I'm cleaning up our backlink profile and looking up anchortexts in OSE. I downloaded and selected one anchortext. However when I go to the sites OSE found, I can't find the links. I look in the source code and onpage keywords. Is it because of my lack of skills that I can't find the links 🙂 or isn't OSE working properly.
Link Building | | StephWeigert0