Disallow: /404/ - Best Practice?
-
Hello Moz Community,
My developer has added this to my robots.txt file: Disallow: /404/
Is this considered good practice in the world of SEO? Would you do it with your clients?
I feel he has great development knowledge but isn't too well versed in SEO.
Thank you in advanced,
Nico.
-
Thank you Lesley.
This really helps a lot. I appreciate it very much. This is my site by the way: http://devilswink.com/
Thanks.
Nico.
-
This comes down to personal preference in my opinion. I think honestly it is neither here nor there. The instances that your 404 page could come up in the SERP's is more than likely pretty low and at the same time it really does not offer any useful content. So disallowing it would not really be any loss. One reason why it might be disallowed is that you have an e-commerce site that rotates products. When a product is deleted, the developer has a 301 to the 404 page, then with the robots.txt saying not to index the 404 page, the other page will drop out of search engines. If this is the case I would rethink that strategy. If you notice a lot of sites like amazon and other big sites leave the page in the index even if the product is no longer for sale. The thought is traffic is traffic, the hardest part in the whole equation is getting someone to your site, if the page is ranking, why delete it.
The only time I can think that I would specifically allow it and optimize it is if you have a cool 404 page. Some companies actually spend a bit of time on their pages and it gets them a little pop of viral traffic from social sharing sites like reddit. If you do have one that is funny or unique I would allow it and actually optimize it for a term like "funny 404 page" or something like that.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Structured data / Rich snippets
I have made FAQ to my website Lån utan UC. I have double checked several time and the code is correct. THe problem is that the FAQ it does not show up i Google search. I have put the code at several subpages, like for example: Mikrolån and at all other sites it shows in SERP but not for the first mentioned site above. That site is the startpage/homepage is that relevant to the issue? Please help and I would be really happy. I have tried to fix this in months...
Technical SEO | | LanutanUC0 -
Internal links best practices
In looking at the inbound links to a client’s Home page, I see that the link from each page of the website back to the Home page is an image, and the ALT text is “Home.” I have a few questions about this, and would appreciate help understanding best practices: --Does it matter that the link back to the Home page is an image (presumably the client’s logo)? -- If we keep the image link, wouldn’t it be better to use “client’s company name” as ALT text rather than “Home”? --Should I recommend using an HTML link back to the Home page, and using the company name as anchor text? (I don't think it's relevant, but the site is built in Drupal.) Thanks!
Technical SEO | | jrae0 -
Image Height/Width attributes, how important are they and should a best practice site include this as std
Hi How important are the image height/width attributes and would you expect a best practice site to have them included ? I hear not having them can slow down a page load time is that correct ? Any other issues from not having them ? I know some re social sharing (i know bufferapp prefers images with h/w attributes to draw into their selection of image options when you post) Most importantly though would you expect them to be intrinsic to sites that have been designed according to best practice guidelines ? Thanks
Technical SEO | | Dan-Lawrence0 -
Product Level 301 Redirects Best Practice
When creating a 301 mapping file for product pages, what is best practice
Technical SEO | | Bucktown
for which version of the URL to redirect to? Base directory or one
subdirectory/category path. Example Old URL: www.example.com/clothing/pants/blue-pants-123 Which of the following should be the new target URL: www.example.com/apparel/pants/blue-pants-123 www.example.com/apparel/blue-apparel/blue-pants-123 www.example.com/apparel/collections/spring-collection/blue-pants-123 www.example.com/blue-pants-123 This is assuming the canonical tag will be www.example.com/blue-pants-123. Also, if www.example.com/blue-pants-123 cannot be reached via site
navigation would it be detrimental to make that the target URL if Google
cannot crawl that naturally? Thanks0 -
How to handle outdated products/pages?
Hi, I wonder how I should handle pages with outdated products. Our market is changing rapidly unfortunately so products we offered in the past are no longer existent. So for example one of our product was called "PRODUCT 1" The URL for this product is still:
Technical SEO | | bastelele
www.mydomain.com/product-1/ with different subpages about the product like:
www.mydomain.com/product-1/about/
www.mydomain.com/product-1/howto/
www.mydomain.com/product-1/pricing/
www.mydomain.com/product-1/whatever/ Our new product is totally different from the first one. Is it still a good idea to set up 301 redirects or are there any other ideas? Thanks in advance
Bastian0 -
Googlebot does not obey robots.txt disallow
Hi Mozzers! We are trying to get Googlebot to steer away from our internal search results pages by adding a parameter "nocrawl=1" to facet/filter links and then robots.txt disallow all URLs containing that parameter. We implemented this late august and since that, the GWMT message "Googlebot found an extremely high number of URLs on your site", stopped coming. But today we received yet another. The weird thing is that Google gives many of our nowadays robots.txt disallowed URLs as examples of URLs that may cause us problems. What could be the reason? Best regards, Martin
Technical SEO | | TalkInThePark0 -
What's the best canonicalization method?
Hi there - is there a canonicalization method that is better than others? Our developers have used the
Technical SEO | | GBC0 -
Best Way to change domain name
Hi, I have a driving instructor wordpress website called www.drivnglessonsbrightonandhove.co.uk, done a little link building. But before I start this properly, I want to change it to www.drivingbrighton.co.uk (as the first is too long to fit on the car etc!!!!) Whats the best way to do this? Is there a way to help keep its position in google as just got into top10 for different keywords?
Technical SEO | | Ant710