Do you add 404 page into robot file or just add no index tag?
-
Hi,
got different opinion on this so i wanted to double check with your comment is.
We've got /404.html page and I was wondering if you would add this page to robot text so it wouldn't be indexed or would you just add no index tag? What would be the best approach?
Thanks!
-
Hello Rubix,
Saijo gave you some great advice, but I'm concerned about the fact that you have that page in the first place, and that it produces those URL parameters. It suggests to me that instead of showing a 404 error on the contact-office.aspx page (assuming that pages doesn't exist on that URL) you are redirecting the user who tries to access that URL to the /404.html page (e.g. /404.html?aspxerrorpath=/contact-office.aspx).
Typically you want the 404 http status code to show on the URL the user is trying to unsuccessfully access. In this case instead of redirecting them to your "404 page URL" you would want to show your customized 404 message (and ensure it returns a 404 status code, use this tool) on www.yourdomain.com/contact-office.aspx.
I hope this makes sense to you. If not, feel free to ask for clarification.
-
404 are OK on your site just make sure you send the proper 404 header response for the 404 page ... Google does NOT index 404 pages ( as long as it sends the 404 header response ) , so you don't need to block them via robots.txt or meta robots.
Infact GWT warns you about these if they are able to crawl the so called 404 pages that doesn't send a 404 header response , so I think its a good idea NOT to noindex them you will get the warning if something is wrong.
Google will only index your 404 if you don't do that..they call it soft 404 : https://support.google.com/webmasters/answer/181708?hl=en
worth reading : http://moz.com/learn/seo/http-status-codes
-
Thanks Martijn,
I actually want to know what would you do for the 404 page itself. It is something like:
www.mainurl.com/404.html and for some reason this started to create some other links such as
www.mainrul.com/404.html?aspxerrorpath=/contact-office.aspx
Do you think I should add 404 page and subpages to Robot.txt ?
Thanks!
-
Hi Sida,
I would add a noindex to the page and as you also will return the 404 status code this is enough data for Google to tell not to index the page itself.
Hope this answers your question.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I index resource submission forms, thank you pages, etc.?
Should I index resource submission forms, thank you, event pages, etc.? Doesn't Google consider this content too thin?
Intermediate & Advanced SEO | | amarieyoussef0 -
H1 Page Title Tag Placement
I'm a little confused over the correct area to place a H1 title Tag. When I look at Wordpress templates and published Wordpress sites, it suggests placing the H1 tag within the header area. However, SEO companies and other well postioned sites place the H1 title tag at the start of the main content area. What is the correct and/or best practice for placing H1 page title tags? Thanks Mark
Intermediate & Advanced SEO | | Mark_Ch0 -
Drop in indexed pages!
Hi everybody! I've been working on http://thewilddeckcompany.co.uk/ for a little while now. Until recently, everything was great - good rankings for the key terms of 'bird hides' and 'pond dipping platforms'. However, rankings have tanked over the past few days. I can't point my finger at it yet, but a site:thewilddeckcompany.co.uk search shows only three pages have been indexed. There's only 10 on the site, and it was fine beforehand. Any advice would be much appreciated,
Intermediate & Advanced SEO | | Blink-SEO0 -
Our login pages are being indexed by Google - How do you remove them?
Each of our login pages show up under different subdomains of our website. Currently these are accessible by Google which is a huge competitive advantage for our competitors looking for our client list. We've done a few things to try to rectify the problem: - No index/archive to each login page Robot.txt to all subdomains to block search engines gone into webmaster tools and added the subdomain of one of our bigger clients then requested to remove it from Google (This would be great to do for every subdomain but we have a LOT of clients and it would require tons of backend work to make this happen.) Other than the last option, is there something we can do that will remove subdomains from being viewed from search engines? We know the robots.txt are working since the message on search results say: "A description for this result is not available because of this site's robots.txt – learn more." But we'd like the whole link to disappear.. Any suggestions?
Intermediate & Advanced SEO | | desmond.liang1 -
Which index page should I canonical to?
Hello! I'm doing a routine clean up of my code and had a question about the canonical tag. On the index page, I have the following: I have never put any thought into which index path is the best to use. http://www.example.com http://www.example.com/ http://www.example.com/index.php Could someone shed some light on this for me? Does it make a difference? Thanks! Ryan
Intermediate & Advanced SEO | | Ryan_Phillips1 -
Why should I add URL parameters where Meta Robots NOINDEX available?
Today, I have checked Bing webmaster tools and come to know about Ignore URL parameters. Bing webmaster tools shows me certain parameters for URLs where I have added META Robots with NOINDEX FOLLOW syntax. I can see canopy_search_fabric parameter in suggested section. It's due to following kind or URLs. http://www.vistastores.com/patio-umbrellas?canopy_fabric_search=1728 http://www.vistastores.com/patio-umbrellas?canopy_fabric_search=1729 http://www.vistastores.com/patio-umbrellas?canopy_fabric_search=1730 http://www.vistastores.com/patio-umbrellas?canopy_fabric_search=2239 But, I have added META Robots NOINDEX Follow to disallow crawling. So, why should it happen?
Intermediate & Advanced SEO | | CommercePundit0