Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do you find popular blog topics (methodology) to rank in Google (organically)
Hi guys Do you guys have any advice and experience into how to find very popular blog topics? The purpose is to be able to rank them organically in Google. If you could share your methodology I would be really thankful as well if possible. Cheers John
Moz Pro | | igniterman750 -
Discrepancy between backlinks (opensiteexplorer vs. search console)
Hey there, when I check in search console for "Links to your site" and "your most linked content" I see incoming links pointing to the wrong (country and language) subdirectory? When I check those same backlinks with opensiteexplorer they seem to point (after redirects) to the correct subdirectory. Anyone out there with similar experiences or thoughts on this? I would be happy to give access to search console if you wanted to have a look at this. Cheers, Jochen When I check in search console for "Links to your site" and "your most linked content" I see incoming links pointing to the wrong (country and language) subdirectory? When I check those same backlinks with opensiteexplorer they seem to point (after redirects) to the correct subdirectory. Anyone out there with similar experiences or thoughts on this? I would be happy to give access to search console if you wanted to have a look at this.
Moz Pro | | Online-Marketing-Guy
Cheers,
Jochen0 -
New Google AdWords Tool: How will this change your KWR?
Now that Google is debuting its new Google Adwords Keyword tool, how will this affect the way you do keyword research? Are there other tools you use?
Moz Pro | | alhallinan0 -
Is there any way to export the organic traffic data?
All of the other Moz pages allow us to export for reports...why can I not export the organic traffic data page? I want a one stop shop...I don't want to have to go to Analytics! Don't make me leave the Moz systemmmmm! 🙂
Moz Pro | | Jeannie.0 -
Creating a campaign and connecting it to Google Analytics
I want to create three campaigns in Moz and connect them to my Google Analytics account. However, two of the domains I am trying to create campaigns for are joined in Google Analytics. Do I need to separate the two in Google in order to create separate campaigns for each in Moz?
Moz Pro | | ERKSPAL0 -
OSE - Group by Domain Export
When exporting inbound link data from OSE, it seems to lose the "group by domain" filter. Is it possible to download the data like that or do I need to filter through all the links myself? Thanks!
Moz Pro | | cccreationsinc0 -
Why am I getting different ranking results on Google?
Because the rank checker on SEOMOZ is temporarily down for upgrade, I followed the link over to the other suggested rank checker on seobook.com which installs a plug-in to Firefox and it seems to be working fine….BUT my question is when I check out my two highest traffic keywords which are “Whitby holiday cottages” and “Whitby cottages” against my website www.endeavourcottage.co.uk it tells me I'm ranking position 2 on both Google.com and Google.co.uk but I don't think I am. When I do the same check using a proxy server called hidemyass.com to check my rankings again on Google.co.uk and .com I appeared to be lower down the rankings like position 8. Does anyone have any idea of which is the accurate reference to my rankings? What do you do to check your rankings ? And one last quick secondary question is when I searching Google chrome using same key words again I come at the top position A on the Google maps listing but using Firefox or Internet Explorer there is no Google maps visible ways that? At the end of the day out rankings are important but where are working towards more traffic...... Cheers
Moz Pro | | whitbycottages0 -
Drastic Difference between Ranking in Seomoz Keyword Ranking and non-personalized Google US Search
Hi everybody, for some of our keywords we saw a huge drop in ranking in the Seomoz Dashboard, yet when performing a non-personalized Google US search we still hold our old positions. Now, I am worried whether this is an upcoming Google penalty or just some bug in the latest crawl. Did anybody ever experience something similar and how could we find out if we were indeed penalized by Google (as we did some major linkbuilding campaign for these keywords recently)? Cheers and a major thanks for your suggestions, Tobias
Moz Pro | | space560