How long after a URL starts showing a 404 does Google stop crawling?
-
Before hiring me to do SEO, a client re-launched their site and did not 301 the old URLs to the new. Only the home page URL stayed the same.
For a month after the re-launch, the old URLs returned a 404. For the next month, all 404 pages (basically any non-existent URL) were 301'd to the home page. Finally, 2 months after launching, they properly 301'd the old URLs to the new.
Now, the new URLs are not ranking well. I assume it's too late to realize any benefit from the 301's, just checking to see if anybody has any insight into how long Google keeps trying to crawl old/404/improperly 301'd URLs.
Thanks!
-
Thanks for the responses. I tend to agree, but my concern is the "permanent" nature of a 301. I assume they will re-crawl the old URLs and replace the previous redirects with the new ones eventually.
-
Agree. Google will follow the links. Keep checking webmaster tools and 301 the pages Google can't find. I can't see why you would not get the link value.
-
Google's crawlers follow links. As long as links exist, those links will be crawled, even if the link generates a 404 error. They could choose to follow known 404 links less frequently, but they will still make attempts to follow the link.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I analysis my domain (http://yourdigibuddy.com/) on moz domain analysis. But there is not data show and show message "No data here". What should I do?
_ I analysis my domain (http://yourdigibuddy.com/) on Moz domain analysis. But there is not data show and show the message "No data here". What should I do?_
On-Page Optimization | | MelissaDooley0 -
Url shows up in "Inurl' but not when using time parameters
Hey everybody, I have been testing the Inurl: feature of Google to try and gauge how long ago Google indexed our page. SO, this brings my question. If we run inurl:https://mysite.com all of our domains show up. If we run inurl:https://mysite.com/specialpage the domain shows up as being indexed If I use the "&as_qdr=y15" string to the URL, https://mysite.com/specialpage does not show up. Does anybody have any experience with this? Also on the same note when I look at how many pages Google has indexed it is about half of the pages we see on our backend/sitemap. Any thoughts would be appreciated. TY!
On-Page Optimization | | HashtagHustler1 -
Keyword in URL: Ranking Factor?
I've got a site about a specific topic, which we'll call "themes" for the sake of this discussion. I personally like to keep the url structure short and clean (for usability purposes, but mainly because I'm a perfectionist and a minimalist). I feel that adding "themes" to the url structure is a bit redundant. However, nearly every keyword phrase that my site should rank for includes the word "themes." So I'm wondering how much I'm handicapping myself by not including the keyword "themes" in the url? The domain name itself sort of includes the keyword . . . although it's in Italian (I chose the domain for it's brand-ability, not for the keyword). A quick example: My Url Structure: www.themo.com/topic/abc My Competitor's Url Structure: www.sitesample.com/themes/topic/abc For many of the keywords, the competitors with the keyword in the url rank highest. But, I'm not sure how much emphasis to place on this, because from my understanding Google doesn't pay as much attention to url keywords anymore . . . and those sites might just be ranking high because they've been around for so long (which also happens to be the reason why they coincidentally also include the keyword in the url, because they started the site when that was a high ranking factor). Thoughts? Should I just trash my perfectionism and add the keyword to the url structure? (By the way, the site is only a couple months old and doesn't have any significant backlinks to inner pages yet, so changing the url structure wouldn't be a big deal if I decided to do that).
On-Page Optimization | | JABacchetta0 -
Google cached snapshots and last indexed
My question is I noticed today that the snap shots of my main pages were outdated. About a month. Then I clicked on the "Learn More" link about cahced images and Google says "Google crawls the web and takes snapshots of each page. When you click Cached, you'll see the webpage as it looked when we last indexed it." I know this sounds really dumb, but does that really mean the last time Google indexed that page? So the changes I have made since then have not been taken yet?
On-Page Optimization | | cbielich0 -
New Url Structure
Hey Guys We are working on a new site and in order to implement some of the new functions we need to restructure our url's , Will redirect everything correctly but I was looking for advice on the structure we need the word product / category subfolder for speed but would there be any benefit making them shorter ? what would you guys advise ? Category Current http://www.freestylextreme.com/uk/Home/Brands/DC-Shoe-Co-/default.aspx **New ** freestylextreme.com/uk/category/dc-shoe-co Product Current http://www.freestylextreme.com/uk/Home/Brands/DC-Shoe-Co-/Mens-DC-Shoe-Co-T-shirts/DC-Black-Unwind-T-Shirt---.aspx New freestylextreme.com/uk/product/dc-black-unwind-t-shirt
On-Page Optimization | | elbeno0 -
301 redirect and then keywords in URL
Hi, Matt Cutts says that 301 redirects, including the ones on internal pages, causes the loss of a little bit of link juice. But also, I know that keywords in the URL are very important. On our site, we've got unoptimized URLs (few keywords) in the internal pages. Is it worth doing a 301 redirect in order to optimize the URLs for each main page. 301 redirects are the only way we can do it on our premade cart For example (just an example) say our main (1 of the 4) keywords for the page is "brown shoes". I'm wondering if I should redirect something like shoes.com/shoecolors.html to shoes.com/brown-shoes.html In other words, with the loss of juice would we come out ahead? In what instances would we come out ahead?
On-Page Optimization | | BobGW0 -
How do I fix a 404 with a 301
I understand the need for fixing 404's but I have yet to have a serious walkthrough of how to set up a 301. From all the talk on the forums and such I'm pretty sure this is easy but I've just never done it before and I could really use a walk through. Thanks
On-Page Optimization | | BenRWoodard1