Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Google could quickly fix the whole Links problem...
A Thursday morning brainstorm that hopefully an important Google manager will see... Google could quickly end all the problems of link buying, spammy links, and negative SEO with one easy step: Only count the 100 best follow links to any domain. Ignore all the nofollows and everything beyond the 100 best. They can choose what "best" means. Suddenly links would be all about quality. Quantity would not matter. Fiverr links, comment links, and all the other mass-produced spam links would literally be ignored. Unless that's all a domain had, and then they would surely be stomped by any domain with 100 decent natural links. Would it be an improvement over today's situation?
Industry News | | GregB1230 -
Are large property portals going to continue to dominate Google's search results?
We are having a discussion (potential argument) in our office around whether large portals (namely property portals) have longevity in Google's search. Argument 1:
Industry News | | NeilPursey
Google's rise in local search and rewarding strong brand names rather than keyword driven domain names will devalue property portals with keyword rich domain names. Property portals are essentially duplicating content on smaller individually owned property websites, therefore in time Google will devalue property portals. **Argument 2: **
Property portals have more property stock listed on their websites so therefore Google will reward them by ranking these websites higher than the smaller real estate agencies with niche stock in their areas that they operate in. The property portals that already are already in a dominate position already carry authority and their own sense of branding, therefore it's difficult for Google to ignore them. If we assume that Google is looking into user behaviour as a ranking factor, then this will help portals as they have more stock which means higher engagement on the website. I'd love to read the moz community thoughts and opinions on this. I reckon it's a worthy debate..0 -
Spanish equivalent of MOZ? need help with understanding Google.es
I try to keep up to date with everything SEO to ensure my sites rank well.. marblerenovation.com & Trailersafe.co.uk I now have a spanish site that I am having trouble ranking well for clima2000egara.es I have recently found out (by asking our lovely moz community) that googles algorithms are not rolled our world wide. SO I have no idea where I am at with Google.es. Does anyone know how I can find out at what stage google.es is at? and what the last update was? and whether there is an equilalent community like Moz for Spain? I am fluent spanish so the language won't be a problem.. Thanks in advance for all your help
Industry News | | david.smith.segarra0 -
Is there a way to get a list (backlink profile) of all tiny url's that point to my site or a competitors site?
I have noticed that most all links you find in all the major back link profile tools such as OSE or GWM, etc... do not show tiny url's. If there is a service that shows all the tiny urls pointing to your site, can someone please share. It has already been proven that tiny url's do pass link juice, so with that being said... if there is no way to find all the tiny urls that point to a site, wouldn't it be a great strategy to create all my back links with tiny url's to mask my profile from competitors? Thanks!
Industry News | | johnd57890 -
Need a contractor to create Wikipedia pages
Hey guys! Can anyone recomend a good contractor to create/maintain Wikipedia pages? We are in the publishing business and I want to create Wikipedia pages for our authors/products. Need someone who has successfully created wikipedia pages before, can make basic research to find sources that wikipedia will consider reliable, has good academic writing skills and can start a debate in case they want to remove our article. If anyone knows good contractors please recommend. Thanks!
Industry News | | Alexey_mindvalley0 -
Questions to help get to know a potential client and see what the possibilities are for there campaign?
OK so I am putting together a list of questions to refer potential clients to. The purpose of these questions is to: 1, Help them understand the difference between "just give us your keywords and we will make you rank" seo and the full range of options that are advocated here on seo moz. .... 2, Get them to see that there are different levels of involvement that are available for different aspects of a campaign and give them ideas and options on how much input they want to have as well as make them see that we can help them with even the stuff they are gonna do themselves or do it for them with there input and permission. Thats where things get a bit confusing I was thinking of having a list of fifty linkbuilding/linkbaiting techniques and check box options beside each one so that they can read a little bit about the particular technique ie forum participation for example and then see how they felt like that related to there industry , wether they would want there staff to do it with some training from us wther they would want us to help them find the right forums and show them how to link back and to which page, or wether they would just want us to handle everything or simply not feel that that was right for there business .. then they would move on to the next thing ie video creation or user generated content and in the end we would have a big list of things to discuss with them and they would have a better understandiing of how much there is to good seo and what might work for them and also how we could help them and how a relationship/ partnership might look I would love some feed back on this is it possible will it help us to build trust with potential clients (which is diffucult for a new biz in this industry) how would you lay it out would it be a usefull resource for other seo companies etc thanks so much seo mozzers this forum is awsome and i know you will steer me in the right direction oh ps this would be something we would keep on our site
Industry News | | duncan2740 -
Any pointers for my site would be greatly apprecieated and rewarded.
We just started changing all the meta data and adding relevant blog content. Anything I'm missing? Any advise or "after panda" link building strategies? inkfarm.com
Industry News | | ibex0