404s effecting crawl rate?
-
We made a change to our site where we all of a sudden we are creating a large number of 404 pages. Is this effecting the crawl/indexing rate?
Currently we've submitted 3.4 million pages, have over 834K indexed but have over and 330K pages not found. Since the large increase in 404s we've noticed a decrease in pages crawled per day. I found this Q & A in Webmasters (http://googlewebmastercentral.blogspot.com/2011/05/do-404s-hurt-my-site.html) but it seems like the 404s should not have an effect. Is this article out of date?
What do you think fellow Moz-ers? Is this a problem?
-
It's not a problem, just fix those as soon as you can. And yes, it does affect crawl rate from what I've seen.
-
That article you mention is very up to date. but if you got "hit" by Google bot several times a day for those pages that now you return a 404 response code you will see a decrease in pages crawled per day since once Google sees a 404 response code it will not visit / hit that page that often aftre that...
-
Yes i've seen this numerous of times. Is it just 404's are are there also things like DNS playing along?
But if the amount of 404's jump up really high then for sure google turns down the speed. i guess this gives you some air to fix it in time.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Question about a Screaming Frog crawling issue
Hello, I have a very peculiar question about an issue I'm having when working on a website. It's a WordPress site and I'm using a generic plug in for title and meta updates. When I go to crawl the site through screaming frog, however, there seems to be a hard coded title tag that I can't find anywhere and the plug in updates don't get crawled. If anyone has any suggestions, thatd be great. Thanks!
Technical SEO | | KyleSennikoff0 -
If I change Tags and Categories in Wordpress blog post, will it negatively affect SEO and cause 404s?
Hi, I have belatedly come to the conclusion that I have been using tags and categories when blogging in wordpress incorrectly. The result is that Google seems to prefer to show my archives and tags in search results rather than the post itself. Not good UX. As the site is only a few months old, am I best to learn my lesson and tag and categorize correctly moving forward or Should I go back in to these posts and clean them up & categorize and tag correctly. If I do this, will it cause 404s and hurt my SEO? Thanks!
Technical SEO | | johnyfiveisalive2 -
404 crawl errors ending with your domain name??
Hello, I have a crawl test with numerous 404 errors ending with my domain name..? Not sure what the cause is. Plugins? Ecommerce? I use Wordpress if that could lead to an answer. Thanks for your time. K
Technical SEO | | Hydraulicgirl0 -
Will presentational HTML tags actually be having a negative SEO effect?
Hi All, I have been using some scanning tools as usual and I keep getting notified about presentational HTML. I'm not involved in the web design but if this is actually having a negative effect then we need to get it changed. Can anyone advise? "pages of this website were found to use presentational HTML elements and/or attributes. It is widely regarded that use of presentational HTML like and should be avoided."
Technical SEO | | SanjidaKazi0 -
During my last crawl suddenly no errors or warnings were found, only one, a 403 error on my homepage.
There were no changes made and all my old errors dissapeard, i think something went wrong. Is it possible to start another crawl earlyer then scheduled?
Technical SEO | | KnowHowww0 -
Question about duplicate content in crawl reports
Okay, this one's a doozie: My crawl report is listing all of these as separate URLs with identical duplicate content issues, even though they are all the home page and the one that is http://www.ccisolutions.com (the preferred URL) has a canonical tag of rel= http://www.ccisolutions.com: http://www.ccisolutions.com http://ccisolutions.com http://www.ccisolutions.com/StoreFront/IAFDispatcher?iafAction=showMain I will add that OSE is recognizing that there is a 301-redirect on http://ccisolutions.com, but the duplicate content report doesn't seem to recognize the redirect. Also, every single one of our 404-error pages (we have set up a custom 404 page) is being identified as having duplicate content. The duplicate content on all of them is identical. Where do I even begin sorting this out? Any suggestions on how/why this is happening? Thanks!
Technical SEO | | danatanseo1 -
How does Google Crawl Multi-Regional Sites?
I've been reading up on this on Webmaster Tools but just wanted to see if anyone could explain it a bit better. I have a website which is going live soon which is going to be set up to redirect to a localised URL based on the IP address i.e. NZ IP ranges will go to .co.nz, Aus IP addresses would go to .com.au and then USA or other non-specified IP addresses will go to the .com address. There is a single CMS installation for the website. Does this impact the way in which Google is able to search the site? Will all domains be crawled or just one? Any help would be great - thanks!
Technical SEO | | lemonz0 -
How to increase the crawl rate?
hello, Our site was hosted in North America and Google was crawling it reasonably fast. Since our traffic is mostly from India we moved it to India, now the crawling is terribly slow from Google. Is there anyway to fix the crawl rate(we have increased the crawl rate in GWT)
Technical SEO | | greyniumseo0