How long after a URL starts showing a 404 does Google stop crawling?
-
Before hiring me to do SEO, a client re-launched their site and did not 301 the old URLs to the new. Only the home page URL stayed the same.
For a month after the re-launch, the old URLs returned a 404. For the next month, all 404 pages (basically any non-existent URL) were 301'd to the home page. Finally, 2 months after launching, they properly 301'd the old URLs to the new.
Now, the new URLs are not ranking well. I assume it's too late to realize any benefit from the 301's, just checking to see if anybody has any insight into how long Google keeps trying to crawl old/404/improperly 301'd URLs.
Thanks!
-
Thanks for the responses. I tend to agree, but my concern is the "permanent" nature of a 301. I assume they will re-crawl the old URLs and replace the previous redirects with the new ones eventually.
-
Agree. Google will follow the links. Keep checking webmaster tools and 301 the pages Google can't find. I can't see why you would not get the link value.
-
Google's crawlers follow links. As long as links exist, those links will be crawled, even if the link generates a 404 error. They could choose to follow known 404 links less frequently, but they will still make attempts to follow the link.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Page Titles in Crawl Errors (although Google is rewriting in serps ??)
Hi Im working on a client/project and crawl report is showing thousands of dupe page titles In the case of the blog/news section its aprox 50 since aprox 50 posts and they all have the same meta-title: "Brand News | Brand" as opposed to: "Title Unique to Page/Topic/KW Relating to Content | Brand" Since these are the main content pages we want to rank (in addition to the main site category pages) then i have instructed dev must prioritise populating these pages meta-titles with the actual post/article titles, as per the latter version of the above example. (I should mention that i have requested they fix all dupe titles but main content pages are the priority). Whilst this will reduce the number of dupe titles in crawl error/warning report which is a good thing, is it actually likely to increase the ranking of these news/content pages given that Google does seem to be rewriting the titles correctly in the serps based on the page content ? Many Thanks in advance for your input
On-Page Optimization | | Dan-Lawrence0 -
Sorry, but that URL is inaccessible
Our site is all on the https protocol so every time I use the on-page grader it tells me the link is unavailable. What can we do? When I use the http protocol (which is 301 redirected to the https) it still gives me the same message.
On-Page Optimization | | whiskeyfl1 -
Handling a Huge Amount of Crawl Errors
HI all, I am faced with a crawl errors issue of a huge site (>1MiO pages) for which I am doing On-page Audit. 404 Erorrs: >80'000 Soft 404 Errors: 300 500 Errors: 1600 All of the above reported in GWT. Many of the error links are simply not present on the pages "linked from". I investigated a sample of pages (and their source) looking for the error links footprints and yet nothing. What would be the right way to address this issue from SEO perspective, anyway? Clearly. I am not able to investigate the reasons since I am seeing what is generated as HTML and NOT seeing what's behind. So my question is: Generally, what is the appropriate way of handling this? Telling the client that he has to investigate that (I gave my best to at least report the errors) Engaging my firm further and get a developer from my side to investigate? Thanks in advance!!
On-Page Optimization | | spiderz0 -
Crawl Diagnostics - Duplicates and canonical problem
SEOmoz crowl diagnostic reports duplicates (title, content) issue on this addres: http://www.meblobranie.pl/biurowe/fotele-biurowe/promocje page already has canonical tag - is this a bug of crowler, or smth wrong on page?
On-Page Optimization | | SITS0 -
Html and css errors - what do SE spiders do if they come across coding errors? Do they stop crawling the rest of the code below the error
I have a client who uses a template to build their websites (no problem with that) when I ran the site through w3c validator it threw up a number of errors, most of which where minor eg missing close tags and I suggested they fix them before I start their off site SEO campaigns. When I spoke to their web designer about the issues I was told that some of the errors where "just how its done" So if that's the case, but the validator still registers the error, do the SE spiders ignore them and move on, or does it penalize the site in some way?
On-Page Optimization | | pab10 -
Google Maps API as primary navigation
Is it okay for SEO to have a google maps api as the primary source of navigation? For example, have people find locations on a map instead of links to them. I'm wondering how/if Google views this method, kinda like how Google can't read images. Will Google realize that these pages are linked to from the homepage gmaps API?
On-Page Optimization | | terran0 -
Pages that show in Open Site Explorer that show as 404's in search
When I use Site Explorer to find our Top Pages there are a whole series of pages like this: http://www.nile-cruises-4u.co.uk/mybestbets/mybestbets-culture.html which I think were created when we had a Bulletin Board on the site. Although they are showing in the Top Pages results when you visit the page you get the following error message: HTTP Error 404.0 - Not Found The resource you are looking for has been removed, had its name changed, or is temporarily unavailable. What implications result from these pages still showing in results and what action should we take without messing up our search rankings? Any advice would be gratefully received.
On-Page Optimization | | NileCruises0 -
All Caps in URL
Hello, we're working with a corporate client to make changes to their URL structure. We recommended that they use a structure like "domain.com/state/city/location". Their IT department is on board, but they just mentioned that all of the state and city info will be pulled from a database where it is all caps. So it would be like this "domain.com/STATE/CITY/location" I'm concerned that it may be spammy, but I can't find any definitive information online. We usually like to test issues like this on our own sites before advising clients, but we're working up against a quick deadline. Any help with this issue would be great, especially real world experience, not just theories. Thank you.
On-Page Optimization | | interactivek0