404s effecting crawl rate?
-
We made a change to our site where we all of a sudden we are creating a large number of 404 pages. Is this effecting the crawl/indexing rate?
Currently we've submitted 3.4 million pages, have over 834K indexed but have over and 330K pages not found. Since the large increase in 404s we've noticed a decrease in pages crawled per day. I found this Q & A in Webmasters (http://googlewebmastercentral.blogspot.com/2011/05/do-404s-hurt-my-site.html) but it seems like the 404s should not have an effect. Is this article out of date?
What do you think fellow Moz-ers? Is this a problem?
-
It's not a problem, just fix those as soon as you can. And yes, it does affect crawl rate from what I've seen.
-
That article you mention is very up to date. but if you got "hit" by Google bot several times a day for those pages that now you return a 404 response code you will see a decrease in pages crawled per day since once Google sees a 404 response code it will not visit / hit that page that often aftre that...
-
Yes i've seen this numerous of times. Is it just 404's are are there also things like DNS playing along?
But if the amount of 404's jump up really high then for sure google turns down the speed. i guess this gives you some air to fix it in time.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Submitted URL has crawl issue - Submitted URL seems to be a Soft 404 - but all looks fine
Google Search Console is showing some pages up as "Submitted URL has crawl issue" but they look fine to me. I have set them as fixed but after a month they were finally re-crawled and google states the issue persists. Examples are: https://www.rscpp.co.uk/counselling/175809/psychology-alcester-lanes-end.html
Technical SEO | | TommyNewmanCEO
https://www.rscpp.co.uk/browse/location-index/889/index-of-therapy-in-hanger-lane.html
https://www.rscpp.co.uk/counselling/274646/psychology-waltham-forest-sexual-problems.html There's also some "Submitted URL seems to be a Soft 404": https://www.rscpp.co.uk/counselling/112585/counselling-moseley-depression.html I also have more which are "pending", but again I couldn't see a problem with them in the first place. I'm at a bit of a loss as to what to do next. Any advice? Thanks in advance.0 -
Blocked URL parameters can still be crawled and indexed by google?
Hy guys, I have two questions and one might be a dumb question but there it goes. I just want to be sure that I understand: IF I tell webmaster tools to ignore an URL Parameter, will google still index and rank my url? IS it ok if I don't append in the url structure the brand filter?, will I still rank for that brand? Thanks, PS: ok 3 questions :)...
Technical SEO | | catalinmoraru0 -
Salvaging links from WMT “Crawl Errors” list?
When someone links to your website, but makes a typo while doing it, those broken inbound links will show up in Google Webmaster Tools in the Crawl Errors section as “Not Found”. Often they are easy to salvage by just adding a 301 redirect in the htaccess file. But sometimes the typo is really weird, or the link source looks a little scary, and that's what I need your help with. First, let's look at the weird typo problem. If it is something easy, like they just lost the last part of the URL, ( such as www.mydomain.com/pagenam ) then I fix it in htaccess this way: RewriteCond %{HTTP_HOST} ^mydomain.com$ [OR] RewriteCond %{HTTP_HOST} ^www.mydomain.com$ RewriteRule ^pagenam$ "http://www.mydomain.com/pagename.html" [R=301,L] But what about when the last part of the URL is really screwed up? Especially with non-text characters, like these: www.mydomain.com/pagename1.htmlsale www.mydomain.com/pagename2.htmlhttp:// www.mydomain.com/pagename3.html" www.mydomain.com/pagename4.html/ How is the htaccess Rewrite Rule typed up to send these oddballs to individual pages they were supposed to go to without the typo? Second, is there a quick and easy method or tool to tell us if a linking domain is good or spammy? I have incoming broken links from sites like these: www.webutation.net titlesaurus.com www.webstatsdomain.com www.ericksontribune.com www.addondashboard.com search.wiki.gov.cn www.mixeet.com dinasdesignsgraphics.com Your help is greatly appreciated. Thanks! Greg
Technical SEO | | GregB1230 -
Pagination/Crawl Errors
Hi, Ive only just joined SEO moz and after they crawled my site they came up with 3600 crawl errors mostly being duplicate content and duplicate urls. After researching this it soon became clear it was due to on page pagination and after speaking with Abe from SEO mozhe advised me to take action by getting our developers to implement rel=”next” & rel=”prev” to review. soon after our developers implemented this code ( I have no understanding of this what so ever) 90% of my keywords I had been ranking for in the top 10 have dropped out the top 50! Can anyone explain this or help me with this? Thanks Andy
Technical SEO | | beck3980 -
Has any positive or negative effect for the SEO results if the domain contains desired keyword?
Helo! Has any positive or negative effect for the SEO results if the domain contains desired keyword? Thanks for the answer.
Technical SEO | | Brainsum0 -
Lots of overdynamic URL and crawl errors..
Just wanted some advice. SEOmoz crawl found out about 18,000 errors. The error URLs are all mainly URLs like the one below, which seem to be the registration URL with a re-direct on, going back the product after registration: http://www.DOMAIN.com/index.php?_g=co&_a=reg&redir=/index.php?_a=viewProd%26productId=3465 We have the following line in the robots file to stop the login page from being crawled: Disallow: /index.php?act=login If I add the following, will it stop the error? Disallow: /index.php?act=reg Thanks in advance**.**
Technical SEO | | filarinskis0 -
How much effect does number of outbound links have on link juice?
I am interested in your thoughts on the effect of number of outbound links (obls) on link juice passed? ie If a page linking to you has a high number of obls, how do you compute the effect of these obls and relative negative effect on linkjuice. In the event that there are three sites on which you have been offered the opportunity of a link Site A PA 30 DA50 Obls on page 10 Site B PA 40 DA50 Obls on page 15 Site C PA 50 DA50 Obls on page 20 How would you appraise each of these prospective page links (ignoring anchor text, relevancy, etc which will be constant) Is there a rule of thumb on how to compare the linkjuice passed from a site relative to its PA and the number of obls? Is it as simple as page with 10 obls passes 10x juice of page with 100 obls?
Technical SEO | | seanmccauley0 -
Honeypot Captcha - rated as "cloaked content"?
Hi guys, in order to get rid of our very old-school captcha on our contact form at troteclaser.com, we would like to use a honeypot captcha. The idea is to add a field that is hidden to human visitors but likely to be filled in by spam-bots. In this way we can sort our all those spam contact requests.
Technical SEO | | Troteclaser
More details on "honeypot captchas":
http://haacked.com/archive/2007/09/11/honeypot-captcha.aspx Any idea if this single cloaked field will have negative SEO-impacts? Or is there another alternative to keep out those spam-bots? Greets from Austria,
Thomas0