Why is 4XX (Client Error) shown for valid pages?
-
My Crawl Diagnostics Summary says I have 5,141 errors of the 4XX (Client Error) variety. Yet when I view the list of URLs they all resolve to valid pages.
Here is an example.
http://www.ryderfleetproducts.com/ryder/af/ryder/core/content/product/srm/key/ACO 3018/pn/Wiper-Blade-Winter-18-Each/erm/productDetail.doThese pages are all dynamically created from search or browse using a database where we offer 36,000 products.
Can someone help me understand why these are errors.
-
We had spaces in the URL that browsers handled well but not spiders. We replaced the spaces with dashes with dynamic code and...it's off to the next problem.
Thanks
-
I think something is going on with a space vs a %20
- Copy and paste in the url that you listed above to Brent's recommended tool, you get a 404 response.
- If you copy and paste that url to a browser, however, your page comes up.
- Now take THAT url and paste into the tool Brent recommends, and you get a 200 (good) response.
The only difference that I see, is that when I copy and paste that url to a browser (Chrome in my case), it adds a %20 where you have a space.
Since this is the thing that makes these other url checkers work, I am guessing that the crawl diagnostics tool is having a similar problem. See the comparison below (much abbreviated to the area in question)
ACO 3018 (from your post, and gives you the error)
ACO%203018 (when it resolves in the browser, and shows a good response in the tools)
I am just smart enough to tell you that these are different, but not smart enough to know why it causes problems for crawlers, but not for browsers.
The good news is that your pages work for users. The bad news is that Google probably never sees them.
-
You need to look at the pages like a spider would. Here is a great tool to check and view the server response. In this case, you will need to go to your developers and allow them to look at this as well.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Just validating a theory I have about why Google is Soft 404ing some of my doctor pages.
As of June 1 doctor pages on our website that say "No ratings are available yet" are being Soft 404ed in our Google Console. We suspect the issue is that wording, due to this post. https://www.contentkingapp.com/academy/index-coverage/faq/submitted-soft-404/ Just wondering if anyone with more expertise than me on 404s or local seo can validate that it is likely this issue. Some examples:
Technical SEO | | Patrick_at_Nebraska_Medicine
https://www.nebraskamed.com/doctors/neil-s-kalsi
https://www.nebraskamed.com/doctors/leslie-a-eiland
https://www.nebraskamed.com/doctors/david-d-ingvoldstad 63647547-aa09-42a8-afe5-4431f277f611-image.png0 -
Why are crawlers not picking up these pages?
Hi there, I've been asked to audit a new subdomain for a travel company. It's all a bit messy, so it's going to take some time to remedy. However, one thing I couldn't understand was the low number of pages appearing in certain crawlers. The subdomain has many pages. A homepage, category pages then product pages. Unfortunately, tools like Screaming Frog and xml-sitemaps.com are only picking up 19 pages and I can't figure out why. Google has so far indexed around 90 pages - this is by no means all of them, but that's probably because of the new domain and lack of sitemap etc. After looking at the crawl results, only the homepage and category (continent pages) are showing. So all the product pages are not. for example, tours.statravel.co.uk/trip/Amsterdam_Kings_Day_(Start_London_end_London)-COCCKDM11 is not appearing in the crawl results. After reviewing the source code, I can't see anything that would prevent this page being crawled. Am I missing something? At the moment, the crawl should be picking up around 400+ product pages, but it's not picking up any. Thanks
Technical SEO | | PeaSoupDigital0 -
Duplicate Page Errors
Hey guys, I'm wondering if anyone can help... Here is my issue... Our website:
Technical SEO | | TCPReliable
http://www.cryopak.com
It's built on Concrete 5 CMS I'm noticing a ton of duplicate page errors (9530 to be exact). I'm looking at the issues and it looks like it is being caused by the CMS. For instance the home page seems to be duplicating.. http://www.cryopak.com/en/
http://www.cryopak.com/en/?DepartmentId=67
http://www.cryopak.com/en/?DepartmentId=25
http://www.cryopak.com/en/?DepartmentId=4
http://www.cryopak.com/en/?DepartmentId=66 Do you think this is an issue? Is their anyway to fix this issue? It seems to be happening on every page. Thanks Jim0 -
Rel canonical for partner sites - product pages only or also homepage and other key pages?
Hello there Our main site is www.arenaflowers.com. We also run a number of partner sites (eg: http://flowershop.cancerresearchuk.org/). We've relcanonical'd the products on the partner site back to the main (arenaflowers.com) site. eg: http://flowershop.cancerresearchuk.org/flowers/tutti_frutti_es_2013 rel canonicals back to: http://www.arenaflowers.com/flowers/tutti_frutti_es_2013). My question: Should we also relcanonical the homepage and other key pages on partner sites back to the main arenaflowers website too? The content is similar but not identical. We don't want our partner sites to be outranking the original (as is the case on kw flower delivery for example). (NB this situation may be complicated by the fact we appear to have an unnatural link penalty on af.com (and when we did an upgrade a while back, the af.com site fell out of the index altogether due to some issues with our move to AWS.) We're getting professional SEO advice on this but wondered what the Moz community's thoughts were.. Cheers, Will
Technical SEO | | ArenaFlowers.com0 -
Errors - 7300 - Duplicate Page Content..Help me..
Hi, I just received the crawl report with 7300 errors of duplicate page content. Site built using php. list of errors will be like this.. http://xxxxx.com/channels/ http://xxxxx.com/channels/?page=1 http://xxxxxx.com/channels/?page=2 I am not good in coding and using readymade script for this website. could anyone guide me to fix this issue? Thanks.
Technical SEO | | vilambara0 -
How to fix this 404 : Error ( 4XX (Client Error) )
In my report this indicates 404 : Error http://www.thexxxhouse.com/what_sets_us_aparat.html This web page removed from server .How to fix this in SEO friendly way .
Technical SEO | | innofidelity0 -
What's the difference between a category page and a content page
Hello, Little confused on this matter. From a website architectural and content stand point, what is the difference between a category page and a content page? So lets say I was going to build a website around tea. My home page would be about tea. My category pages would be: White Tea, Black Tea, Oolong Team and British Tea correct? ( I Would write content for each of these topics on their respective category pages correct?) Then suppose I wrote articles on organic white tea, white tea recipes, how to brew white team etc...( Are these content pages?) Do I think link FROM my category page ( White Tea) to my ( Content pages ie; Organic White Tea, white tea receipes etc) or do I link from my content page to my category page? I hope this makes sense. Thanks, Bill
Technical SEO | | wparlaman0 -
Do you validate you websites?
Do you consider the guidelines from http://validator.w3.org/ when setting up a new website? As far as I know they don't influence rankings ... What is your opinion about that topix?
Technical SEO | | petrakraft0