Tons of Crappy links in new OSE (Open Site Explorer)
-
I am starting to miss the old OSE. I've found that for a lot of the pages on our site, the new OSE is showing WAY more links and most of them are garbage nonsense links from China, Russia, and the rest of the internet Wild West.
For instance, in the old OSE, this page used to show 9 linking domains:
http://www.uncommongoods.com/gifts/by-recipient/gifts-for-him
It now shows 454 links. Some of the new links (about 5 of them) are legitimate. The other 400+ are garbage. Some are porn sites, most of them don't even open a web page, they just initiate some shady download. I've seen this for other sites as well (like Urban Outfitters) This is making it much harder for me to do backlink analysis on bc I have no clue how many "Normal" links they have. Is anyone else having this problem ? Any way to filter all this crap out ? See attached screenshot of the list of links I'm getting from OSE.
-
Ok thank you. I will email directly.
-
Hey Zack,
Sorry to hear you're still having problems - we've seen an improvement on most sites at this point. Would you want to send me info on the site you're searching and any filters you are using?
If you don't feel comfortable posting that info on this thread, feel free to email me directly: [email protected].
Thanks!
Carin
-
Hey Carin,
I just wanted to follow up on this...I'm still seeing these spammy binary files show up as links. Unfortunately it makes OSE quite useless for me in regards to exploring our own backlinks.
What is the status of this problem? Has there been any headway ? Why does our site have problems but most others don't?
Thanks!
-Zack
-
Hey Zack,
Thanks so much for understanding! We are doing everything we can to get the bug resolved. Binary files are the downloadable files you see as links - .pdf, .exe, .img, etc.
I'm really sorry, but we don't have a URL to the old OSE. I saw Steven's response as a workaround - is that possible or are there too many file types to filter out?
Our crawlers that provide the metrics to OSE are always crawling, but will take about a month for our fix to propagate through to all the pages we crawl. Once we have removed these links from our crawlers, then we'll have to process the metrics. This is why it's looking like late September for the fix to show up.
I really appreciate your patience and understanding, we're doing everything we can to fix it!!
Thanks,
Carin
-
Hey Carin-
Thank you so much for this in-depth response. Glad to hear that you guys are aware of it and trying to sort it out. Very interesting info...I'd never hear of "binary" links before but I hope you guys can figure out how to handle these. Seems like a tough task to tackle, just by looking at my CSV it looks like these come in several different forms and they could be hard to identify..I have a few questions:
1. Is there by chance a URL you could give me that points to the old OSE ?
2. How often does OSE crawl? Is it a constant process or are there scheduled crawls?
Thanks!!
-Zack
-
Hey Zack, I saw the ticket you filed was answered by Aaron, but I just wanted to follow up with you as well. We have made some really exciting changes to the crawler, but, unfortunately, there is a pretty obvious bug as well...
The reason for the “questionable” links coming from the Internet Wild West is due to the crawler reaching much deeper into sites where there are more download (i.e. binary) links. The first issue is the crawler is counting a binary file as a link, but the larger issue, is that the crawler doesn’t really know how to handle these types of files. This bug is causing some links to be improperly associated with certain domains. This is probably what you're seeing with all the crazy links from China and Russia which don't actually link to the site you're researching.
There are two steps to addressing this issue: changing how the crawler sees these file types and then fixing how the crawler handles these file types. We have made improvements to our algorithm so that we will be handle the majority of these files correctly, however, this update will need about a month to propagate. The fix for this issue probably won’t be seen for two more updates, meaning late September. Our improvements should catch most of the issues, but there still could be a few cases we haven't addressed. If this happens, don't hesitate to let us know; we love feedback since it helps us improve and make our index even better!
The next step is to fix how our crawlers handle binary file links and prevent them from being improperly associated with certain domains. We are in the process of working through that issue right now. We’re doing everything we can to resolve this bug as we know it is alarming to see these “questionable” links associated with your sites.I hope this helps and thanks so much for being patient :)Thanks,Carin
-
2 ways:
- Get as CSV and spend the time going through it
- Wait it out
-
OK cool good info, hope they fix it soon!! Any good ideas on how you can filter this crap[ out ?
-
Hello Zack,
That is an issue that they are working on, I know this because I already discussed this with one of their help desk people. Here is the page that describes the changes: http://www.seomoz.org/blog/brand-new-open-site-explorer-is-here
In addition to that, here is some additional information I can share with you:
you may see “questionable” links with weird file extensions. This is due to the crawler reaching much deeper into sites where there are more download links. We are looking into fixing this bug as soon as we can so these won’t be counted as links.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New to Moz, need some probably basic answers about Keywords, Linking, Competitors and General SEO
Hi, So I have quite a lot of data colelcted about my site now, regarding keyword research, page crawling and competitor research ect. But I find myself second guessing myself about what I have done and what to do next. I have done basic research for as many relevant keywords I could think of to my site, including branded and non branded terms. If the main competitive keywords for my niche are very competitive, shall I start doing more research for long tail keywords and only try to rank for them? Does is matter how many keywords I am doing research for? Does is matter how many keywords I try to optimise for each webpage? Are the amount of branded keywords I am researching skewing my results? As they are all ranked #1, but nearly all of the non branded keywords are much further down the list... Once I have decided what keywords are worth trying to ranking for for each page, are the techniques to actually rank more highly for them - Title, H1 Tag, Description, Meta Data, Fresh Content and using the keywords on the page? Or are there more techniques I haven't heard of? Under Keyword Rankings - I noticded that some of my keywords are directing to specific pages, like "Cavity Waxes" is directing to the URL ending in .com/cavity-waxes - How do you assign the keywords im researching to specific URLs? - Or does Moz do it automatically? As most of my keywords seem to be unassigned to any URL, is that because they are not ranking highly enough? How do I best use the data collected through Moz? Good practices? Techniques? Tips and Tricks? What is the best practice for finding potential link partners and asking them for mutual linking? Techniques for finding partners that are likely to link with us, but still provides link juice. I must apologise for this long-winded set of questions, but these are troubling me! Any help would be greatly appreciated, Kind regards, Max Johnson
Moz Pro | | BiltHamber10 -
Opensite Explorer
It may sound like a stupid question but what kind of likes and shares is Opensite Explorer referring at? Is it just how much your page gets liked and shared or about the posts on FB?
Moz Pro | | BridalHotspot0 -
Competitive Link Comparison vs. Open Site Explorer
I get different external followed links showing in the Competitive Link Comparison vs. Open Site Explorer. The total number of links is different as well. I thought this was the same info - just one tracks ongoing and one is just a "one-off" tool. Is this not the case? Do the count links differently? Thanks! Kevin
Moz Pro | | yandl0 -
Seomoz and site explorer completely different
seomoz and site explorer completely different results on own site and competitors on moz i am ahead in all aspects in site explorer i am way off par...why
Moz Pro | | lordmoose85450 -
Internal Links
I've searched high and low, but I could just be missing it... Is there a report on internal links? Maybe a count? Or better yet, a way to see internal anchor text?
Moz Pro | | MCIMaui0 -
Sites Blocking Open Site Explorer? Penguin related.
Last week I was looking at a competitors site who has a link scheme going on and I could actually check the links for each anchor text. This week they don't work at all, do you think they're blocking the rogerbot on their domains? Or is there a problem with open site explorer? http://www.opensiteexplorer.org/anchors?site=www.decks.ca If you're interested in the background, all the links are to instant-home-biz . com which then redirects decks . ca - it's a tricky technique. Pretty much all of the links are from sketchy sites like: airpr23.xelr8it.biz/ airpr23.anzaland.net/ airpr23.vacation-4-free.com/airpr23.blogfreeradio.net/airpr23.blogomatik.com/http://www.morcandirect.com/mortgages/resources2.php which I thought Penguin was supposed to catch…
Moz Pro | | BeTheBoss0 -
Alexa Ranking Sites
I found these two sites giving my competitor link juice: http://www.webnamelist.com/alexa/Alexa_186.html http://www.list-of-domains.org/alexa/Alexa_185.html I have seen these sites before and I just dont get why they are authoritative. The funny thing is I did a search for my competitors link on the page and its not showing up, is this a problem in site explorer? Why is site explorer mentioning these sites as my competitions best links when these links do not exist on their site?
Moz Pro | | SEODinosaur0 -
Y!SE and OSE
Hi So now when Y!SE is closed im trying to find alternative, so im trying OSE basically i like the data that it showed to me but the numbers is unbelievable different. Im used Y!SE about 1 year so i have statistic for a year. Previous month for my client homepage i have 2625 total external links and now OSE reports me only 1558, for the main category page was 16 now 68\. Most strange for me is total external links for root domain on Y!SE its always been around 6-7k but now OSE reports me 10,053. So my question why numbers so different can it be that the Y!SE don`t update for several month there data or some others reason? Thanks, Sergey
Moz Pro | | 77Agency0