Cannot crawl website with redirect intalled on subdomain url
-
Hi!
I want to crawl this website : http://www.car-moderne.ch.
I tried a got back the crawl just for that one url (not for all the pages of the website). This single line cvs says that the status of the http://www.car-moderne.ch is 200, but in fact it is a redirect 301 to http://www.car-moderne.ch/fr where the live home page is (actually the Moz bar sees the 301, not the 200 as the single-lined crawl does).
How can I proceed in this case (a 301 redirect being installed on the subdomain url) to still be able to have a full-fledged juicy cvs with all the broken links, duplicate content, etc.
Thank you for your help!
Pascal Hämmerli
-
So glad to help, Pascal!
-
Dear Chiaryn,
Thank you for your very helpful reply.
This website is hosted on a partner agency who create the website and I only act as a SEO consultant for them. What you say is very helpful because it means their home-made CMS should be corrected to provided better 301 redirection.
I wish you a good day,
Pascal
-
Hey Pascal,
Sorry for the confusion here! It looks like the subdomain, www.car-moderne.ch, returns a 200 HTTP status to our crawler and to other crawlers, such as the hurl.it tool. In the body of the screenshot I attached from the hurl.it tool, the only code there is the number 404, so basically the site is serving a page with no crawlable data. The page isn't redirecting and it doesn't return any real source code, so there is no data for us to include in the crawl. I would recommend working with your webmaster to resolve this issue and to get the page to correctly serve a 301 redirect to the /fr version of the site to all crawlers.
I can see that the site is correctly responding with a 301 redirect for some crawlers, such as this test I ran as googlebot, but the response doesn't seem to be consistent. One thing you will want to be sure to have your webmaster check is how the site responds to user-agents that are hosted on Amazon Web Services, as some of our crawlers and the hurl.it crawl are both hosted through AWS.
Once the issue of the HTTP response is resolved, you should be able to get much better data from the crawl test tool.
I hope this helps! Please let me know if I can help you with anything else.
Chiaryn
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl the site anymore
Hi Moz I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl anymore. When I look at the crawl report there is just one fault displayed: "403 : Received 403 (Forbidden) error response for page." How can i solve this issue asap? I was not aware that there would be any issues with moz when activating a SSL certificate on the site. The campaign with the problem is: ramxpert.dk. Regards
Moz Bar | | WebBoost
Jesper Nielsen0 -
Is there a way to export all your crawl errors for multiple Moz campaigns at once?
We're looking for a simple way to export all crawl errors for our Moz campaigns. More than likely we could use the API, but was wondering if there was any functionality already built into Moz for exporting all crawl errors.
Moz Bar | | ReunionMarketing0 -
Has anyone had to deal with Moz crawl issues on their Zendesk support site?
If so - how did you end up resolving them? For instance we have 85 "temporary redirect" errors from our Zendesk support site in our crawl error report and we don't have access to the robots.txt file through Zendesk.
Moz Bar | | zspace0 -
Duplicate page found with MOZ crawl test?
When I crawl my website www.radiantguard.com, the crawl test comes back with what appears to be a duplicate of my home page: http://www.radiantguard.com and http://www.radiantguard.com/ Does the crawler indeed see two different pages and therefore, are my search engine rankings potentially affected, AND is this because of how my rel canonical is set up?
Moz Bar | | rhondafranklin0 -
Site Crawl report show strange duplicate pages
Beginning in early in Feb, we got a big bump in duplicate pages. The URLs of the pages are very odd: Example URL:
Moz Bar | | Neo4j
http://[email protected]/dir/page.php
is duplicate with http://website.com/dir/page.php I checked though the site, nginx conf files, and referral pages, and could not find what is prefixing the pages with 'http://firstname.lastname@'. Any ideas? The person whose name is 'Firstname Lastname' is stumped as well. Thanks.0 -
URL Link Counts
Hi Can someone clarify what the URL link counts in the SERP report means? Is this total internal links on this page? My followed internal links are very high compared with sites I am comparing against, some sites only have 1 internal followed link - is this possible? Have they simply no followed all the rest on that page?
Moz Bar | | BeckyKey0 -
Pinterest Back Links Showing For One Website But Not Other in OSE
Hello, I have verified my business websites on both of my pinterest accounts for the following websites Http://www.prickettproperties.com and Http://www.ChicFamilyTravels.com The backlinks for prickett shows up in OSE, but not for ChicFamily. I am curious as to why that is? Thanks in advance for your help
Moz Bar | | Prickett0 -
Why isn't seo moz properly crawling my site?
In my campaign results, only 2 pages have been crawled, when there are many more. Also, when I do the on-page evaluation for my homepage and the main keyword, I get a grade of F, with a ton of errors that my page actually does not have. It's as if SEO moz is having trouble crawling through my site. Any ideas on this?
Moz Bar | | diplomajim0