Googlebot encountered extremely large numbers of links on your site??? How Do I resolve this?
-
I am working on a site with over 30 million pages. Every time I get about One Million indexed I get a Message in the Google Webmasters Tools saying "Googlebot encountered extremely large numbers of links on your site"
The indexing then starts dropping like a Rock. I need to get the site indexed. Please Help!
-
Kenneth
I work with extremely large sites quite often. There's no single answer to this because it depends on what's going on as to why the Googlebot is breaking down in its crawl. For example - how many links exist on any single page? Is it 100, 300, 1000 or more? The more links on every page the more likely the bot will choke, though it's a lot better than it used to be.
Does the site validate for markup? Or could there be choke-points due to validation errors?
Is the content organized in an intelligent funnel structure, or is everything one level off the root domain?
Is there only one way for the bot to navigate deep into the site, or are there multiple methods to get down deep?
Is some of the content only linked from within in a way that many of those links are not discovered until the bot has to first go through six or eight other layers, some of which could be timing out just when the bot gets there?
How many quality inbound links point to pages deep within the site?
These are all questions that need to be asked and answered and that's just scratching the surface of the problem potential.
The most important thing is to try and think like the bot - if I go here, will I become overwhelmed? if I go here, will I hit a road-block?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Change Phone Number Based on Traffic Source + Ping URL for Call Tracking Number
Hi Everyone, Is there a tool that can change the phone number on a web page based on the visitor source (i.e., direct, organic, paid, etc.)? I'd like to implement a solution like this with different call tracking numbers based on the visitor source. We use the Google suite for our analytics (GA, GTM, Google Data Studio, Google Optimize is also an option as well). - Also, is there a good call tracking service that will ping a URL each time the phone number is called so that we can track these calls as events in GA? The majority of our visitors use a desktop PC and dial in the number on the screen rather than clicking (tapping) on it from a mobile device. Thanks, Andy
Reporting & Analytics | | AndyRCWRCM0 -
Ecommerce site product link. How to handle a link that doesn't exist.
Suppose we have this product A, and we just have a single item for this. When the item is sold out we do not want to show it on the website saying "out of stock". Instead we would like to remove the product from out store which will now result in a url that doesn't exist. And google webmaster tool and Moz analytic will show them as page not found after they crawl over the site. Should i be generating a new sitemap.xml and update ? How do i handle those pages that don't exist anymore ? Thanks
Reporting & Analytics | | MindlessWizard0 -
Sudden Increase In Number of Pages Indexed By Google Webmaster When No New Pages Added
Greetings MOZ Community: On June 14th Google Webmaster tools indicated an increase in the number of indexed pages, going from 676 to 851 pages. New pages had been added to the domain in the previous month. The number of pages blocked by robots increased at that time from 332 (June 1st) to 551 June 22nd), yet the number of indexed pages still increased to 851. The following changes occurred between June 5th and June 15th: -A new redesigned version of the site was launched on June 4th, with some links to social media and blog removed on some pages, but with no new URLs added. The design platform was and is Wordpress. -Google GTM code was added to the site. -An exception was made by our hosting company to ModSecurity on our server (for i-frames) to allow GTM to function. In the last ten days my web traffic has decline about 15%, however the quality of traffic has declined enormously and the number of new inquiries we get is off by around 65%. Click through rates have declined from about 2.55 pages to about 2 pages. Obviously this is not a good situation. My SEO provider, a reputable firm endorsed by MOZ, believes the extra 175 pages indexed by Google, pages that do not offer much content, may be causing the ranking decline. My developer is examining the issue. They think there may be some tie in with the installation of GTM. They are noticing an additional issue, the sites Contact Us form will not work if the GTM script is enabled. They find it curious that both issues occurred around the same time. Our domain is www.nyc-officespace-leader. Does anyone have any idea why these extra pages are appearing and how they can be removed? Anyone have experience with GTM causing issues with this? Thanks everyone!!!
Reporting & Analytics | | Kingalan1
Alan1 -
How to Detect Links within PDFs
Hi All, I have a funny situation that I would like some advice on handling... There are a handful of domains that were created several years ago in support of an offline to online campaign. These domains are simply vanity domains that use an IFrame at 100% to show the content of another page. Essentially, the content of the sites I manage are embedded into the frame on the vanity URL. Since I do not monitor or have access to any analytics for the vanity URLs, is there a way to tell how others are discovering those vanity URLs? As stated above, they were used on direct mail flyers two years ago and never appeared online. However, I still get a good deal of traffic from them and cannot believe people have hung onto those flyers in such volume. I have used Open Site Explorer for the vanity URLs, which show no links existing anywhere online. I am wondering if the vanity URLs may exist in pdf lists of local businesses that match my category, etc. Is there any way to tell how traffic finds those vanity URLs without analytics or discovered links through link profiling tools?
Reporting & Analytics | | dsinger0 -
Accidental Link not being removed by Google WMT
I operate two sites for a client. One is a local business and one is their national business. I used the same template for both sites (with changes) but accidentally left a link in the footer to the local site. Now the local site is showing 12k backlinks from the national site. I removed the link over 2 weeks ago but it still shows up in Google WMT in the "Links to your Site" section. It goes to a coupon page and not a "targeted" page but 12k links to the local site is 6 TIMES what I had before. My question is: "Is there a way to get Google to remove the link from Google WMT?" More specifically force it. Like I said the link has been removed for over 2 weeks but it still shows up in the Local site's Incoming Links section of WMT. Thanks.
Reporting & Analytics | | DarinPirkey0 -
Is Webmaster Tools Useless as a broken Link Detector?
Buongiorno from yes we still have free parking Wetherby UK!
Reporting & Analytics | | Nightwing
Ok when it comes to detecting broken links I'm getting really frustrated with webmaster tools. Now I'm probably going to end up with egg on my face with this one but here is an example of webmaster tools reporting a broken link which i cant find. http://i216.photobucket.com/albums/cc53/zymurgy_bucket/phantom-broken-links_zpsb74e1246.jpg Having trawled through the code i just cant see the knackered link? Is it a phantom report or is something usefull being detected here? Grazie tanto,
David1 -
Measuring events to external sites
Im having problem measuring click on ads by using events in GA or Jetpack. For example when I checked out yesterday this is what I read: 1. In GA events it says 12 clicks 2. In Jetpack it says 9 clicks But when I look at Referrals to the actual site directly it says 18 clicks Which one is the rights one? I need this because I use this to invoice clients end of month! and it cant be any "maybe".something. cheers, R
Reporting & Analytics | | rrrobertsson0 -
Is there any web analytics tool that let us track number of outgoing clicks (and visits) ?
I just wonder if we can measure outgoing visits from a specific URL with an online tool or not?
Reporting & Analytics | | merkal20050