Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
All ranked pages on Googles SERP only links to home
I got a problem regarding my website called musik.dk, and hope you guys are able to help. I just got my first ranking results from Moz. My question is: All my keywords are linking to the home page, and not the artist page? For examples if I were to search on Rihanna on Google, then when musik.dk appears on the SERP, it only links to the home, and not musik.dk/rihanna.. This problem applies to all the given artists ranked on Googles SERP, it never shows the artist page itself, only links to home. Let me know if you need any information, and I gladly supply, this is kind of frustrating to me.. As a note: all pages besides the home has a PA of 1, and it doesn't really seem to change. UT3OBHr kVp7z38
Moz Pro | | Morten_Hjort0 -
Crawl results on keyword rankings differ from actual search rankings
For the second week in a row, the weekly keyword rankings in my MOZ account show "not in top 50" for most keywords. However, when I check some of my rankings I still broadly have the rankings that I used to have. It seems that the MOZ crawl is experiencing some difficulties. Does anyone have suggestions how to find out what's going on? If I can't rely on the crawl outcomes, then the tool is not very useful anymore!
Moz Pro | | matteis0 -
Your site's pages may be using techniques that are outside Google's Webmaster Guidelines
Hi All The message below I received from google webmaster, please tell me how I solve this problem Dear site owner or webmaster of http://testedfatburners.com/, We've detected that some of your site's pages may be using techniques that are outside Google's Webmaster Guidelines. If you have any questions about how to resolve this issue, please see ourWebmaster Help Forum for support. Sincerely, Google Search Quality Team
Moz Pro | | mkm1040 -
Getting odd results with MOZbar. (Some pages are 0,0,0)
I'm trying to review the Domain aurhotiry, Page Authority, and MozRank & Moztrust for some news websites and I found it odd that may sites will have excellent DA,MT,MR & PA on most of the pages but then when I view one of their blog posts the PA,MR & MT are 0. Here's are two examples Site
Moz Pro | | SheffieldMarketing
http://www.washingtonpost.com/lifestyle Individual Post
http://www.washingtonpost.com/lifestyle/home/checking-in-with-thomas-pheasant/2012/10/30/a2920ed4-1df5-11e2-ba31-3083ca97c314_story.html Site
http://www.philly.com/philly/living/ Individual Post
http://www.philly.com/philly/home/Home_Style_Silver_makes_holiday_decorations_really_shine_.html Does that mean that links from blog posts would not be very benificial? The domain authoity is still crazy high but everything elese is 0. Anyone know why? I'm new to using the Moz bar. Thanks0 -
Why does google not recognize inbound links from bbb.org?
After using site explorer I noticed my companys links for bbb.org are not included. They are for our competitors but not us. Anyone know why? Thanks, Cole
Moz Pro | | coleda0 -
SEOMoz rocks, but is it breaking Google's webmaster guidelines when it comes to check rankings?
Hi all, The Google's webmaster guideline in question reads... "Don't use unauthorized computer programs to submit pages, check rankings, etc. Such programs consume computing resources and violate our Terms of Service. Google does not recommend the use of products such as WebPosition Gold™ that send automatic or programmatic queries to Google." So, when WebPosition Gold is not recommended then how come SEOMoz is acceptable to check ranks for websites belonging to SEOMoz PRO? Do Google neglects breaking of few guidelines by good boys? Thanks in advance.
Moz Pro | | vinodchaudhary0 -
What is mT/mR good for In the keyword tool difficulty full report results?
So the ratio is MozTrust to MozRank, but what is this good for? What can I deduce from this and what can I use it for?
Moz Pro | | MarloSchneider2