Cannot crawl website with redirect intalled on subdomain url
-
Hi!
I want to crawl this website : http://www.car-moderne.ch.
I tried a got back the crawl just for that one url (not for all the pages of the website). This single line cvs says that the status of the http://www.car-moderne.ch is 200, but in fact it is a redirect 301 to http://www.car-moderne.ch/fr where the live home page is (actually the Moz bar sees the 301, not the 200 as the single-lined crawl does).
How can I proceed in this case (a 301 redirect being installed on the subdomain url) to still be able to have a full-fledged juicy cvs with all the broken links, duplicate content, etc.
Thank you for your help!
Pascal Hämmerli
-
So glad to help, Pascal!
-
Dear Chiaryn,
Thank you for your very helpful reply.
This website is hosted on a partner agency who create the website and I only act as a SEO consultant for them. What you say is very helpful because it means their home-made CMS should be corrected to provided better 301 redirection.
I wish you a good day,
Pascal
-
Hey Pascal,
Sorry for the confusion here! It looks like the subdomain, www.car-moderne.ch, returns a 200 HTTP status to our crawler and to other crawlers, such as the hurl.it tool. In the body of the screenshot I attached from the hurl.it tool, the only code there is the number 404, so basically the site is serving a page with no crawlable data. The page isn't redirecting and it doesn't return any real source code, so there is no data for us to include in the crawl. I would recommend working with your webmaster to resolve this issue and to get the page to correctly serve a 301 redirect to the /fr version of the site to all crawlers.
I can see that the site is correctly responding with a 301 redirect for some crawlers, such as this test I ran as googlebot, but the response doesn't seem to be consistent. One thing you will want to be sure to have your webmaster check is how the site responds to user-agents that are hosted on Amazon Web Services, as some of our crawlers and the hurl.it crawl are both hosted through AWS.
Once the issue of the HTTP response is resolved, you should be able to get much better data from the crawl test tool.
I hope this helps! Please let me know if I can help you with anything else.
Chiaryn
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why can I see 404 pages in Google Analytics but nothing in the On-Demand Crawl?
Hello, I'm looking at some Google Analytics data for a website and can see a few 'Page not found's among the Page Titles, looking like these are 404 errors. To get a full list of what's 404-ing so I can get these redirected, the Moz on-demand crawl of the website has come back with no major errors and just a few metadata ones. Does anyone know any potential reasons why the audit has drawn a blank, and is there another way to get a comprehensive list of 404s, as I'm aware the Google Analytics data may not be covering all of them. Thanks very much Becky
Moz Bar | | becky.jenkins0 -
Significant difference with DA scores on Moz Chrome app VS Website Tool
I saw a big difference between DA scores on Moz Chrome app VS Website Tool. Which DA score is the correct one? I personally believe the scores on Moz's site is most accurate. Do you happen to have issues syncing the scored to your Chrome app? Chrome App VS. Moz Research Tool (https://mza.seotoolninja.com/researchtools/ose/)
Moz Bar | | iPrice_Marketing
Moz.com: 92 / 88 Here's a screenshot of the difference: https://ipricegroup-my.sharepoint.com/:i:/p/jeremy_chew/ESy9lzUTC3lOl8o93Sx6LsUB8mYXo8LmYuUj2aa0xLXi1A?e=A6OrXs This was evident in other websites too:
Priceza (priceza.com.my😞 38 / 24
Shopback (shopback.my😞 43 / 41
Cuponation (https://www.cuponation.com.my/😞 27 / 250 -
Moz Pro: Redirect Chain warning given to pages that don't have redirects
When I look up crawl errors for a page, I'm always told the page suffers from redirect chaining. However, when I do a redirect check (in this case, using the Redirect Path Chrome extension), it indicates that my page does not use a redirect. Why would Moz detect redirects, while no other redirect checker resource does? For example, this URL gets Moz's redirect chain warning: https://www.aem.org/news/january-2018/5-reasons-iot-projects-fail/ But there is no redirect associated with this URL.
Moz Bar | | jrichter0 -
Different Errors Running 2 Crawls on Effectively the Same Setup
Our developers are moving away from utilising robots.txt files due to security risks, so e have been in the process of removing them from sites. However we, and our clients still want to run Moz crawl reports as they can highlight useful information. The two sites in question sit on the same server with the same settings (in fact running on the same Magento install). We do not have a robots.txt files present (they 404), and as per Chiaryn's response here https://mza.seotoolninja.com/community/q/without-robots-txt-no-crawling this should work fine? However for www.iconiclights.co.uk we got: 902 : Network errors prevented crawler from contacting server for page. While for www.valuelights.co.uk we got: 612 : Page banned by error response for robots.txt. These crawls were both run recently, and there was no robots.txt present. Not to mention, they are on the same setup/server etc as mentioned. Now, we have just tested this, by uploading a blank robots.txt file to see if it changed anything - but we get exactly the same errors. I have had a look, but can't find anything that really matches this on here - help would really be appreciated! Thanks!
Moz Bar | | I-COM0 -
301 Redirects detected as duplicates
I have 19 pages that are all 301 redirected to the same page. Moz is detected these 19 pages a duplication's of each other. Does anyone know how to solve this issue?
Moz Bar | | Worship_Digital0 -
Why RogerBot can't crawl site https://unplag.com
Hello Please help me to solve the problem. The on-page grader and Crawl Test are not working for Unplag.com website. Both said that they can't access the url. Yes, I've tried different variants like unplag.com, http://unplag.com One more thing - RogerBot was disallowed in robots.txt file. I deleted it from the file a week ago so maybe moz index haven't been renewed.
Moz Bar | | Targeras0 -
Spam score 9/17 and redirect Question
I sat on a .com domain, which name become increasing popular (xyzselfie) for 2 years ... 4 months ago I hired a VA to do a task. A miscommunication made this person submit my domain to the spammiest directories the internet has to offer. Also because of the domain name and the .com a lot of asian or weird sites/things posted links to my site. I have worked on my site for the last 4 months trying to lower my spam score from a 9. I have:
Moz Bar | | onlinegusto
-Disavowed all the sites that pointed to my site.
-Made more internal links
-Tried to make my content thicker
-Included my email and social profiles to the site In the process my competitors site with exact domain name but .net and more authority came on auction, I bought it and I pointed it with a permanent redirect to my site (hoping my site would in time lose its spam score). This site will generate and income by appearing in search and adsense ads. After months of work I'm at a loss what to do. Does the spam score generally take long to drop? Should i try and stop the permanent redirect and direct my .com to the .net domain? Are there experts who can lower my score? Should I look for non spammy directories in its niche and submit my site to them to increase link authority and nofollow links ? Any feedback or insight would be highly appreciated. fFtTOFk0 -
Does anyone have a good article or video on how to read the SEO MOZ crawl report column by column?
I am trying to find a good how-to on how to read and analyze each column of the SEO MOZ crawl report, specifically, the excel sheet it allows you to export. What I'm really trying to get to the bottom of is what the "Yes" indiciates under rel-cononical. If it says "yes," does this mean that the link in question has been canonoicalized?
Moz Bar | | armcwill0