Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Many Backlinks Can One build To Blog's Homepage Daily To Avoid Algorithm Penalties
Hello, Am new to blogging and also new to link building and things are pretty confusing for me right now, I was told by a friend that have spent years in the industry that the maximum number of Backlinks you should build for a domain to stay safe is 2 daily, but am seeing different thing on the internet, so I wanna know the exact number of quality Backlinks one can build to homepage daily to avoid penalties, please I need suggestions from pros. Sergio!
Link Building | | Gabriel50 -
When pitching a whitepaper as Push Content for Link Building, is it ok to give the person I'm pitching a link to a landing page with a form on it?
When pitching a whitepaper as Push Content for Link Building (i.e. pushing out content that my client has created), is it ok to give the person I'm pitching a link to a landing page with a form on it? Or should I create a landing page with the whitepaper included on it? I’m not sure if the client will be ok with this b/c I know they use the whitepaper for sales purposes to gain leads. For example, my pitch email would include a line such as this, "the whitepaper can be found at LINK and I'd love if you could share it with your readers." I think it may be weird/a little wrong to ask a webmaster to include a link on his site to a landing page with a form to get the whitepaper. Does this make sense? What have others done with whitepapers as Push Content for link building?
Link Building | | ArketiGroup1 -
Help! Someone's inserted my link in the footer of another site!
There's a site in my industry where (I'm guessing on flawed SEO advice) all of their pages contain footer links to rival sites – with each page containing a different set of links. One page has a link to my site, which I've just found out from GWT as the link now produces a 404.
Link Building | | Jeepster
Should I
a) ignore it?
b) ask them to replace it with a live link from my site (their site's highly relevant to mine)? or
c) ask them to remove it altogether as no-one wants footer links?0 -
I'm hiring a link building company - need help choosing
I asked this question before and got mixed feedback. Companies vs. Freelancers. Well I have a couple questions here. At a certain point it is definitely worth going in house, but what about the knowledge that we could gain from working with a big agency? Does that really add any value? Can I hire a freelancer and trust him as much as a big agency? or vice versa? Can I hire an agency and trust them as much as a freelancer? I have also been advised to make sure they offer competitive / site analysis as a first step, AND inform you of exactly how many links they plan to build per month, how much time they will dedicate to you and your account, expect turn around times, and light strategy details that they would likely use. Thanks for any help! TA
Link Building | | TylerAbernethy0 -
Is it ok for a web design company to have a branded footer link on their client's sites?
Now I know that in general footer links to your site from another site are bad...this is because they are very often spammy...however I like to think that Google is pretty smart and I am of the opinion that a web design company should be able to link back to their own site. Here's why: If a visitor comes across a site that they love the design of, and they want a new website built...why shouldn't they be able to click through to the web designers site? (as long as the client is happy to link to it of course) I also feel that if there are a whole bunch of high authority/pagerank websites have been designed by a web design company and they therefore have a footer link pointing to them, it's probably a pretty good sign that they're a good web designer. Is it not? In saying this I think that the link anchor text should be branded rather than keywords. For example I usually write "Web Design by Static Shift" I'm interested to hear people's thoughts. Am I being blinded by my bias? Thoughts aside, and onto the facts...what are people's experiences with footer links for a web design company. Do they help or hinder?
Link Building | | Static_Shift3 -
Changing url from non www to www.
basically I have realised i have more page authority and links going to www.onestopmuscle.co.uk instead of onestopmuscle.co.uk. I use wordpress and have no clue how to make sure onestopmuscle.co.uk redirects to www. version. anyone have any ideas? I don't want to mess about with files or i'll most likely make the blog into a disaster.
Link Building | | FLEAR0 -
What's your favourite SEO directory?
We're having a bit of a debate in the office today about SEO directories. We all seem to have our favourites here - Octopedia, AbiLogic and JoeAnt to name a few that have been mentioned. Just wondering what everyone else's favourite directories for SEO are? Which pass the best link juice? Which offer the best value for money? Would love to know everyone's thoughts...
Link Building | | Digirank1 -
Impact of Panda Algorithm Change on Articles Base.
I am a published author and researcher. All my articles are original content and high quality. If I am looking for backlinks by submitting high quality original articles to Articles Base, how will the panda algorithm change affect me? Should I be submitting articles somewhere else? If so, where? I greatly appreciate any suggestions you may have.
Link Building | | DanManCastro0