Www vs non www - Crawl Error 902
-
I have just taken over admin of my company website and I have been confronted with crawl error 902 on the existing campaign that has been running for years in Moz. This seems like an intermittent problem. I have searched and tried to go over many of the other solutions and non of them seem to help.
The campaign is currently set-up with the url http://companywebsite.co.uk when I tried to do a Moz manual crawl using this URL I got an error message. I changed the link to crawl to http://www.companywebsite.co.uk and the crawl went off without a hitch and im currently waiting on the results. From testing I now know that if i go to the non-www version of my companies website then nothing happens it never loads. But if I go to the www version then it loads right away.
I know for SEO you only want 1 of these URLS so you dont have duplicate content. But i thought the non-www should redirect to the www version. Not just be completely missing.
I tried to set-up a new campaign with the defaults URL being the www version but Moz automatically changed it to the non-www version. It seems a cannot set up a new campaign with it automatically crawling the www version.
Does it sound like im out the right path to finding this cause? Or can somebody else offer up a solution?
Many thanks,
Ben.
-
Glad to hear thats all fixed! Though i will say thats a very slow response time for any development / hosting company typically i would expect a maximum response time of 8 hours.. We try to keep it under 2 heh.
But yes, glad thats working for you now
-
The problem was as you anticipated, after spending a few days chasing the party who actually look after it im pleased to say i checked this morning and all is working as expected.
Thanks very much for your help Toby!
-
Thanks Toby, Ive emailed off, I expect a reply to be a couple of days away (what it normally take them) Thanks for the help thus far and ill message back when they do!
-
In that case that would probably be the best place to start. If you want any evidence for a missing A record, heres a DNS checking tool (it currently throws an error because it can't find an A record).
Let me know what they say
-
I dont have access to the DNS, my access is limited to the Magento CMS, and ftp access to the root folder.
I guess its a case of popping an email off to the guys who built the site and control the web space and getting them to address the issue?
-
Ok, so the file is working, thats a good start!
Looks like we need to go back a step in the request process then. Do you have access to the DNS settings for the http://atp-instrumentation.co.uk domain? If so, please could you check if there is an A record set for it? (looks to be registered through Civica UK Ltd - Whois Report)
What i suspect might be the case is that you're missing an A record for atp-instrumentation.co.uk but that there is one set for the www.atp-instrumentation.co.uk.
I've run a couple of tests against the domain DNS and i get nothing back for the non-www address, which is what suggests that we're not even making it as far as your servers.
To set the A record, you'll be looking for something in your control panel for 'DNS settings' or maybe 'Host Records', you should see in there ether an option to select A record settings, or perhaps a dropdown with things like A, AAA, CNAME etc. You need to:
- select 'A'
- In the domain box type: atp-instrumentation.co.uk
- In the IP box, type: 82.118.110.42
Hopefully that makes sense. If your at all unsure, let me know and i'll do what i can to help more specifically. Domain control panels are so different for each provider its difficult to provide direct instructions without knowing what your panel looks like
-
I added the line and it did indeed break the website, got internal service error etc
So it seems the file is working
-
Just a note here, the Redirects do seem to be working, so it looks like the .htaccess file IS being loaded.
Please check anyway, its possible that the redirects are comming from another location if they have been set elseware as well. From a (very quick) look, the file seems to be formatted correctly so no obvious reason for the www redirects to not be working...
-
Hmm ok,
before i do anything else, we need to make sure that the .htaccess file is actually being loaded. To do that, we need to break it for a second.
at the top of the htaccess file, put something like (specifics dont matter here)
THISisInValid666
What we're trying to do is put some invalid text into the htaccess file so that the site breaks when it loads. The idea being that we can confirm that the .htaccess file is actually being used. So if you put that in and the site throws a 500 error (when navigating to it with or without www) we can confirm that changes we make should work.
If the site continues to load without issue then we know that the .htaccess file isnt in use so we need to look at server configureation directly (specifically the AllowOverrides settings)
Once you have confirmed if it does / doesnt break, remove the line again
-
Thanks Toby, here is the entire HTACCESS file with your fix implemented. Doesn't seem to have worked if i go to
http://atp-instrumentation.co.uk with no www it still fails to load
Edited out to shorten convo
-
You're correct, you can make it a little more generic though, without seeing all of your .htaccess file, try this:
Replace:
RewriteCond %{HTTP_HOST} ^companyname.co.uk [NC]
RewriteRule ^(.*)$ http://www.companyname.co.uk/$1 [L,R=301]With:
RewriteCond %{HTTP_HOST} !^www. [NC]
RewriteRule ^ http%{ENV:protossl}://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]This is what could be called a wildcard redirect in that a direct copy paste should work for you with no need to edit. (you dont have to manually add in the correct domain name)
What it does:
- First it checks to see if the requested url has a www in it
- if it does -not- it then runs the rule, otherwise it ignores it.
- The rule first checks for http or https. Then adds in the www. followed by the domain and tld, finally adding the URI (/somepage/page for example).
- the L in square brakets means do not process anything else in the htaccess file
- the R=301 means that it will be a 301 (perminant) redirect.
If that still doesnt work for you, paste up your full .htaccess file, or you can send it to me directly if you'd rather and i'll take another look
-
Thanks Highland,
How do I go about changing this? I believe its to do with the .htacess file.
The website was developed in Magento via an external company who monitor it. Looking in the root folder I can see the htaccess file but it contains a lot lines of code and rewrites that I dont fully understand.
These lines are the one I think could be relevant so far.
############################################
enable rewrites
Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^companyname.co.uk [NC]
RewriteRule ^(.*)$ http://www.companyname.co.uk/$1 [L,R=301]Then there is lots of category pages etc that are 301 redirected
Follow by
RewriteRule ^home http://www.companyname.co.uk/ [R=301,L]Then some more redirects for pages
I know this is specific but is this editable a different way in Magento? Thanks for any help offered i know this is getting more technical
-
You're 100% right. You should have one 301 redirect to the other. While there are some SEO reasons for this (mainly with duplicate content), the best reason is that it's just less confusing to end users to only have one URL to use. If your non-www has trouble loading I would say you need a 301 to the www version.
Moz restricts Top Level Domain (i.e. domain.com) and crawls accordingly. I have some set up with www.domain.com and some with just domain.com. The 301 to the www forces the issue but Moz is smart enough to crawl the proper pages.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Resolving 301 Redirect Chains from Different URL Versions (http, https, www, non-www)
Hi all, Our website has undergone both a redesign (with new URLs) and a migration to HTTPS in recent years. I'm having difficulties ensuring all URLs redirect to the correct version all the while preventing redirect chains. Right now everything is redirecting to the correct version but it usually takes up to two redirects to make this happen. See below for an example. How do I go about addressing this, or is this not even something I should concern myself with? Redirects (2) <colgroup><col width="123"><col width="302"></colgroup>
Technical SEO | | theyoungfirm
| Redirect Type | URL |
| | http://www.theyoungfirm.com/blog/2009/index.html 301 | https://theyoungfirm.com/blog/2009/index.html 301 | https://theyoungfirm.com/blog/ | This code below was what we added to our htaccess file. Prior to adding this, the various subdomain versions (www, non-www, http, etc.) were not redirecting properly. But ever since we added it, it's now created these additional URLs (see bolded URL above) as a middle step before resolving to the correct URL. RewriteEngine on RewriteCond %{HTTP_HOST} ^www.(.*)$ [NC] RewriteRule ^(.*)$ https://%1/$1 [R=301,L] RewriteCond %{HTTPS} !on RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L] Your feedback is much appreciated. Thanks in advance for your help. Sincerely, Bethany0 -
Error got removed by request
hey there, I have website, Which is Doing best on google, But from last 2-3 times i am Getting error in Google Search Console, removed by request., here is screenshot http://prntscr.com/iva5y0 My Question is i have Not Requested Google to Remove My Homepage URL, Then 1. Why I am Getting This warning again & again, ?? 2. Will this Harm My Sites Current traffic & Ranking Status ? 3.What Steps I need Take to stop This warning came again & again ??? Thnx in advance & please Help Out on this Query
Technical SEO | | innovative.rohit0 -
500 - server error
Hi All, A site crawl reveals several server errors (status code 500) about a clients wordpress website. My question: what are the most common causes for server errors and what advice can I give about how to fix them? Thanks in advance,
Technical SEO | | WeAreDigital_BE
Jens0 -
404 vs 410 vs 301
Hi guys, I am managing a real estate website, and obviously we have a LOT of pages detailing each property. As those properties get sold and removed from the website, I'm wondering how best to handle this - I know 404, 410 and 301's are all valid ways to go, but I want to provide the best UX combined with the best SEO effect. My thinking is to customise a 410 page to show the page has been permanently removed, and has a relevant message (rather than a generic 404 message) and shows a search box - possibly pre-populated according to the page they were looking for.
Technical SEO | | LoonyToons
I think this gives a good UX and helps Google to understand the importance of the 000's of pages on our website.
I'd also like to clear property detail 404's as quick as possible to make it easier to see if we have problems elsewhere on the site. Having explained this to our development/SEO agency, they are strongly pushing for 301 redirects or leave as 404.
I think 301's would be the worst for UX, and as explained earlier, the volume of 404's is massive and makes it difficult to see real errors. They seem to think this is a better UX and better for SEO. Just wondering what you guys would recommend?0 -
Nginx 403 and 503 errors
I have a client with a website that is hosted on a shared webserver running on an Nginx server. When I started working on the website a few months ago I found the server was throwing 100s of 403s and 503s and at one point googlebot couldn't access robots.txt. Needless to say this didn't help rankings! Now the web hosting company has partially resolved the errors by switching to a new server and I'm now just seeing intermittent spikes in Webmaster Tools of 30 to 70 403 ad 503 errors. My questions: Am I right in saying there should (pretty much) be no such errors (for pages that we make public and crawlable). Having already asked the web hosting company to look in to this. Any advice on specifically what I should be asking them to look at on the server? If this doesn't work out, does anyone having a recommendation for a reliable web hosting company in the U.S. for a lead generation website with over 20,000 pages and currently 500 to 1000 visits per day? Thanks for the help Mozzers 🙂
Technical SEO | | MatShepSEO0 -
CamelCase vs lowernodash
I'm in the process of reviewing on-site URL structure on a few sites, and I've run into something I can't decide between. I am forced to choose between the two examples: MediaRoom/CaseStudies.aspx (camel case) mediaroom/casestudies (all lower case, mashed, no dashes) I would personally rather see: media-room/case-studies/ However implementing the dashes would require manually re-writing about ~10,000 URLs. Implementing 301s from the existing structure to whatever I choose would be trivial, so there is no concern there. Given the choice between CamelCase and lower-mashed, which would you choose? Why?
Technical SEO | | MRCSearch0 -
Microsite on subdomain vs. subdirectory
Based on this post from 2009, it's recommended in most situations to set up a microsite as a subdirectory as opposed to a subdomain. http://www.seomoz.org/blog/understanding-root-domains-subdomains-vs-subfolders-microsites. The primary argument seems to be that the search engines view the subdomain as a separate entity from the domain and therefore, the subdomain doesn't benefit from any of the trust rank, quality scores, etc. Rand made a comment that seemed like the subdomain could SOMETIMES inherit some of these factors, but didn't expound on those instances. What determines whether the search engine will view your subdomain hosted microsite as part of the main domain vs. a completely separate site? I read it has to do with the interlinking between the two.
Technical SEO | | ryanwats0 -
Tracking a Crawl error
Hi All, If you find a crawl error on your page. How do you find it? The error only says the URL that is wrong but this is not the location. Can i drill down and find out more information? Thank you!
Technical SEO | | wedmonds0