Any tools for scraping blogroll URLs from sites?
-
This question is entirely in the whitehat realm...
Let's say you've encountered a great blog - with a strong blogroll of 40 sites.
The 40-site blogroll is interesting to you for any number of reasons, from link building targets to simply subscribing in your feedreader. Right now, it's tedious to extract the URLs from the site. There are some "save all links" tools, but they are also messy.
Are there any good tools that will
a) allow you to grab the blogroll (only) of any site into a list of URLs (yeah, ok, it might not be perfect since some sites call it "sites I like" etc.)
b) same, but export as OPML so you can subscribe.
Thanks!
Scott
-
Not at all. I guess my feeling here is that there is a sort of untapped social graph defined by blogrolls. If it were simple to harvest them upon visiting a blog (e.g. this blogger recommends...) one could do a stumble-on-steroids approach to a niche.
-
I thought you might be able to use the outbound link scraper to grab the outbound link onto the page. Pop in your URLS of the pages you want to scrape and it will spit out our a list of those domaind and urls. You can take those urls and put them into the contact finder and it will return the contact details for those sites. Combine the two spreadsheets for an epiuc list of blogs to contact for your outreach.
This is obviously for link building rather than subscribing - sorry if I have misunderstood what you were trying to do
-
Hi Keri,
That is a very cool tool, but is overkill for this. It takes far too many steps to accomplish only part of the desired goal of grabbing all blogroll URLs (within the blogroll DIV tag) and exporting the list to a valid OMPL file or URL list.
thanks!
-
nothing I saw there would do this. It looks like it could manage to list all external links, and I suppose you could manually pick the blogroll out of it.
-
Hi there,
Well, Keris response reminded me of this question and the fact that I found a tool for scraping these kind of lists:
Here it is (with some other cool tools) , have fun:
-
Hi Scott,
I'm going through older questions. Did you ever find a tool to do what you wanted to do here?
-
One thing to look at is Outwit Hub for Firefox. It might be able to help with that. It can scrape data from a page and do a lot with it. http://www.outwit.com/products/hub/. Don't know that it meets all of your needs, but I also haven't seen a response with anything better at the moment.
-
Hey Scott,
What a great question and <sigh>I don't have the answer. I am going to back to find out what people come up with here. Surely there is someone that lurks these parts that can throw something together?</sigh>
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disavowing links and disavow tool
Hello, I have had a look on webmaster tools and Moz and can see lots of bad links on Moz only. I am thinking whether to disavow them and was wondering if you could you let me know if it is possible to have a look at your current disavow file? I do not currently know what links were included the previous time. Is there a way to check if there are any links in the file at all? I thought I would raise this question as there is a lot of negative info about the disavow tool.
Moz Pro | | SEM_at_Lees0 -
Tools for Monitoring Hundreds to Thousands of Keywords and Rankings
Hi All, I am in process of doing and SEO overhaul for our five global sites in: US, UK, Canada, Sweden, France I'd like to track hundreds of keywords and rankings per site - I'm talking at least 300-400 keywords each site. Each site has its own country domain with both www and www2 domains. So, I need a keyword tool that will let me track massive amounts of keywords. I know that the Moz Pro tool helps, but we only have 350 keywords on this account. I think on this. Any suggestions on something reliable that will provide good data? I'm sure I can get some budget to purchase something, but I also can't spend too too much money. I'm not looking for a massive analytics package. Right now, I'm concerned mainly with our keyword rankings Thanks in advance!
Moz Pro | | CSawatzky0 -
Site Explorer Results - Linking Domains Tab
I have a competitor who has the following Linking Root Domains numbers on the Linking Domains tab and was wondering if some one could explain what the numbers mean? opera.com - 205,515
Moz Pro | | bridgeway04-34677
dmoz.org - 97,752 prweb.com - 198,092 quantcast.com - 115,748 tinyurl.com - 443,540 jquery.com - 57,886 URL: http://www.opensiteexplorer.org/domains?site=www.verypdf.com%2F Thanks, Brad0 -
Tools that crawl 2 million page sites
Our site is about 2million pages deep, 50% of which is stale content. Yes, I know - OMG #unhygienic. Even if we get approval to get rid of half of it. SEOMoz Pro Elite only crawls 20k deep - what can i do to crawl and diagnose the whole site. Are there any tools anyone can suggest. SEOMoz??
Moz Pro | | ilhaam0 -
To Many Links on site
I've had an issue with to many links on the site. My drop down menu, secondary footer and footer. The report told me that I had 253 links on each page. I then programmed my secondary footer to dynamic and ran a crawl and my links reduced accordingly to 201. Then turned the footer into dynamic and ran a crawl with my links increasing to 1500. This also happened between each phase but en went away. Oddly enough, my domain authority increased as well as other factors in the crawl report. This too many links thing is driving me crazy. Please provide some guidance.
Moz Pro | | CHADHARRIS0 -
Seomoz research tools question for inbound links
What is the best seomoz tool and or indicator within a tool for checking on a link (directory or article site) to evaluate whether or not you would actaully want a link from them? Any way to see if google penalizes them for there tactics or if that can hurt me by them linking to me or whatever? I am at the beginning of a link building mission, but want to ensure that I do it methodically and correctly as possible based on the combines wisdom of this community. Thanks for your help, Steven
Moz Pro | | sfmatthews0 -
Are there plans for an API for the keyword difficulty tool?
I don't know about everyone else, but this would make me one happy pro member!
Moz Pro | | davidangotti0 -
A tool to submit websites in directories
Hello I am looking for a tool to help me to submit websites in directories, something like the yooda tool. http://www.yooda.com/outils_referencement/submit_center_yooda/ This tool seems good no? do you offer something similar at seomoz? or where could I find some similar tools and in which languages is it available?
Moz Pro | | bigtimeseo2