How effective is OSE in crawling press release links?
-
How effective is OSE in crawling press release links?
We have released a few press releases recently (over the last couple of months) and OSE doesn't seem to have found them.
-
Hey There,
That could be a possibility. It is hard to say definitively given the nature of web crawlers. They just crawl links as they see them in random succession, so a lot of factors come into play.
Best,
Nick
SEOmoz -
Our releases have appeared on big sites like the Financial Post.
Is it possible they just get buried under other news so OSE can't find them? I know Google indexes these pages, we get the alerts.
-
Hey there,
Just so you know, here's how we compile our index: - We grab the most recent index. - We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains). - We start crawling from the top down until we've crawled 59,000,000,000 pages (which is about 25% the amount in Google's index).
Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index. Sorry!
We update our Linkscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always in motion. When we need to start processing, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important. You can see our most recently updated schedule here: http://seomoz.zendesk.com/entries/345964-linkscape-update-schedule
Linkscape focuses on a breadth-first approach. Therefore we almost always have content from the homepage of websites, externally linked-to pages, and pages higher up in a site's information hierarchy. However, deep pages that are buried beneath many layers of navigation are sometimes missed and it may be several index updates before we catch all of these.
If our crawlers or data sources are blocked from reaching those URLs, they may not be included in our index (though links that point to those pages will still be available). Finally, the URLs seen by Linkscape must be linked-to by other documents on the web or our index will not include them.
For now, the best thing you can do to help your domain become indexed is to work on link building for links from sites with high mozrank.
Best,
Nick
SEOmoz -
This all depends on where the press releases have been posted.
If you've got the urls of the sites they're on it may be worth looking at these in OSE to see if SEOmoz has them indexed. However, don't forget that the SEOmoz index is not the same as google's. Just because it's not showing on OSE doesn't mean that G hasn't seen it.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I find links on my site
I'm looking to find a certain type of link on my site. A link that we're directing out of the site. We have a lot of subdomains though and I was wondering if there was a way to find all the links on each subdomain without screaming frog them all?
Reporting & Analytics | | mattdinbrooklyn0 -
Difference between External links and anchor text?
What is the exact meaning of external links & anchor text and what is different between these?
Reporting & Analytics | | surabhi60 -
Cannonical Links?
Hi guys, I recently started using Moz Analytics's for my site and it has told me that the vast majority (perhaps all) of my pages have duplicate content because Google will be indexing the version both with and without www. in front of it as seperate domains. I've done some research and have come across a few suggestions of what to do, but i'm not sure which to go with or how to actually implement it. Any help, advice or suggestions would be greatly appreciated! Thanks
Reporting & Analytics | | Sandicliffe0 -
Google Webmaster Tools - When will the links go away!?
About 9 months back we thought having an extremely reputable company build our client some local citations would be a good idea. You definitely know this citation company, but I'll leave names out. Regardless, it's our mistake to cut corners. Google Webmaster Tools quickly picked up these new citations and added them to the links section. One of these citation spawned a complete mess of about 60K+ links on their network of sites through ridiculous subdomains of every state in the country and so many other domain variations. We immediately went into remove mode and had the site's webmaster take down the bad links from their site. This process took about a month for outreach. The bad links (60K+) have not been on the spam site for well over 6 months but GWT still shows them in the "links to your site" section. Majestic, Bing, and OSE only displayed the bad links for a brief time. Why is webmaster tools still showing these links after 6+ months? We typically see GWT update about every 2 weeks, a month tops. Any ideas? Could a changed robots.txt on the bad site prevent Google from updating the links displayed in GWT? We have submitted to disavow, but Google replied with "no manual penalty". We even blasted the bad site with Fiverr links, in hopes that Google would re-crawl them. No luck with anything we do. We have patiently waited for way too long. The rankings for this site got crushed on Google after these citations. How do we fix this? Should we worry about this? Any advice would really help. Thanks so much in advance.
Reporting & Analytics | | zadro0 -
2 days in the past week Google has crawled 10x the average pages crawled per day. What does this mean?
For the past 3 months my site www.dlawlesshardware.com has had an average of about 400 pages crawled per day by google. We have just over 6,000 indexed pages. However, twice in the last week, Google crawled an enormous percentage of my site. After averaging 400 pages crawled for the last 3 months, the last 4 days of crawl stats say the following. 2/1 - 4,373 pages crawled 2/2 - 367 pages crawled 2/3 - 4,777 pages crawled 2/4 - 437 pages crawled What is the deal with these enormous spike in pages crawled per day? Of course, there are also corresponding spikes in kilobytes downloaded per day. Essentially, Google averages crawling about 6% of my site a day. But twice in the last week, Google decided to crawl just under 80% of my site. Has this happened to anyone else? Any ideas? I have literally no idea what this means and I haven't found anyone else with the same problem. Only people complaining about massive DROPS in pages crawled per day. Here is a screenshot from Webmaster Tools: http://imgur.com/kpnQ8EP The drop in time spent downloading a page corresponded exactly to an improvement in our CSS. So that probably doesn't need to be considered, although I'm up for any theories from anyone about anything.
Reporting & Analytics | | dellcos0 -
Find 404 Broken Links
We are looking for tools to help us repair broken backlinks, those with 404 error. We have used the Open Site Explorer tool to create a CSV file from our client's URL. We read the "Fixing Crawl Diagnostic Issues", but don't see how to "find the 404ed URL", nor do we see a "referral column" to scroll to. What steps should we take to locate the broken 404 linkages? What steps can we take to streamline repairs?
Reporting & Analytics | | jamie_netsitemarketing.com0 -
Finding and removing "Bad" Back Links
In the process of trying to figure out where all of the “Bad” backlinks are coming from I used the SEOmoz Site Explorer. I can see the links that may be questionable but am not sure how to determine if these are the issue causing the loss of rank or could it be something else. On Google webmasters they list Siteloki.com as the one with the most links. The count is now at 13,005. (see attached WMT report)
Reporting & Analytics | | rdominey
I first noticed this a month ago, 6,742 links and have tried contacting them with no reply, no results, I have even posted on the site asking to be removed from their listing and not response. Website: www.getyourphotosoncanvas.com I do not understand why this site is not showing up in the Site Explorer link analysis report (See attached)? Could this be some sort of hack or hidden links that Site Explorer does not see? How do I determine if this is real or not, if it is the reason that Google is demoting us? Google says that we are not being manually penalized? 5zAQq Iz9ct0 -
Is the link data from Open Site Explorer in real time or an average?
I just started using Open Site Explorer to track internal and external link data. Is this information given in real time or is it an average over a specified period of time?
Reporting & Analytics | | mequoda0