Crawl depth seems off?
-
I'm reviewing my site crawl data and am seeing some very strange things such as:
- The homepage URL has a listed crawl depth of 2.
- Pages that are featured in the main site navigation (which is present on all pages, including homepage) are ranking at a crawl depth of 3.
What am I missing here? Shouldn't my homepage have a crawl depth of 0 or 1? Why would pages linked directly from my homepage have a crawl depth other than 1? (Single click from homepage to that page)?
Thank you!
-
Hi Samantha,
I set up a new campaign using the https:// version of the site and ran a new crawl, but I'm running into the same issue as before. Perhaps this is a bigger question of how site redirects work? I was under the impression that any large-scale redirects (such as from non-www to www or http to https across all pages) can affect crawl time/load time. Rereading your comment, it sounds like what you're saying is those redirects count as layers of crawl depth, as well. By the same token, I'm assuming any redirects (301's in particular) also add a layer of crawl depth.
So, my larger question then is: how can I maximize crawl depth if my site has been redirected from http to https? Will that "extra layer" of crawling always be there as long as the redirect is in place, or is there a way to compress/expedite how the crawl happens?
Thanks for your input on this!
-
Hi Samantha,
That makes sense, thank you. I'll set up a new campaign tracking with "https://" instead!
-
Hey there,
Sam from Moz's Help Team here!
So the thing to keep in mind when you set up a campaign at the root domain level is that we'll be starting the crawl from the http protocol (non-www). In this case - http://logic2020.com/. If you filter by crawl depth in your Site Crawl you'll see that URL with a crawl depth of 0.
It redirects to http://www.logic2020.com/ which has a crawl depth of 1. That URL then redirects again to https://www.logic2020.com/, which is listed with a crawl depth of 2 - hence why links we found on that page have a crawl depth of 3.
I hope this helps to clarify but let me know if you have any other questions!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My company has about 70 live pages. Moz crawled 6k. Why?
My company has about 70 live, published pages on our website, which is managed through HubSpot. When Moz crawled the URL, it found 6K+. I noticed several were URLs with a campaign tag. Is this normal? It seems excessive to me. If there's a problem, how do I fix it?
Link Explorer | | ActionableResearch1 -
Crawl Errors on a Wordpress Website
I am getting a 902 error, "Network Errors Prevented Crawler from Contacting Server" when requesting a site crawl on my wordpress website, https://www.systemoneservices.com. I think the error may be related to site speed and caching, but request a second opinion and potential solutions. Thanks, Rich
Link Explorer | | rweede0 -
How many pages Google crawl in free version
I am using moz pro 30 days trial version.Can Anybody tell me how many pages moz crawl in a day or in a week.Because its two days and only 2 pages they crawled. Thanks
Link Explorer | | VarinderS0 -
I crawled my site, but an old crawl report still is visible
I crawled my site recently, but an old crawl report still is still all I can see
Link Explorer | | Bigjim0 -
Spam Score and crawling of my site
Hello, I'm trying to analyze the spam score of my site which is 9/17 Actually I have few backlinks and all of them have a low spam score (max 4/17, just one). I think there's some kind of issue with the crawler since I get strange spam factors: Large Site with Few Links (likely true, I recently deleted a lot of tags used once) Low Number of Pages Found (wasn't it a "Large Site"??) Low Number of Internal Links (I got a considerable number) No Contact Info (I have a link to my facebook in the menu and a "contacts" page) Thin Content (It's just a blog with min 300 words per post, why thin?) Site Link Diversity is Low (likely true) Ratio of Followed to Nofollowed Subdomains (likely true) Low MozTrust or MozRank Score (true) Ratio of Followed to Nofollowed Domains (likely true) Can you please help me to understand it, is it a crawling problem or similar? If needed I will post the url of the website. Thank you so much Marco
Link Explorer | | MarcoBP0 -
Moz can't crawl domain due to IP Geo redirect loop
Hi, I'm trying to crawl our domain www.salvationarmy.org.au via my Moz account and it only ever returns results for one page when it should be crawling more than 3,000 pages. In talking to support, they have said that because of the redirect we have in place it is creating a 302 loop and therefore not delivering results. Usually in this case I would obtain Moz's IP addresses and add them to the redirect settings as an exception, but Moz have said they use cloud-based services for crawling so the IPs change all the time. Does anyone have any idea how to solve this issue? At this point I've paid for a year's subscription to a product I can't use. Thanks, Mel
Link Explorer | | SalvationArmy0 -
Does OSE crawl our site often? Or do we have an other problem? (internal links)
I am wondering if i can find out when OpenSiteExplorer crawled our entire website for the last time.
Link Explorer | | wilcoXXL
A few months ago we added some internal textlinks to all of our productpages. Those are links to the brandpage of that particular product. Some of our brands do have up to 1500 products, so there should be at least 1500 internal links pointing to that brandpage. But OSE still gives me a wrong count. It says there is only 1 internal link pointing to that brandpage.
Is it because OSE never crawled our site again? Or am i missing something here?(maybe too many internal links to one specific brandpage is not okay?) I'll hope you guys can help me out with this one.0 -
Moz crawling bot
Hi guys, in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt". I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot). However, my robots.txt doesn't have such entry. Take a look: Automatically banned scanners and crawlers section User-agent: 008 Disallow: / user-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: metajobbot Disallow: / User-agent: Exabot Disallow: / User-agent: Ezooms Disallow: / User-agent: fyberspider Disallow: / User-agent: dotbot Disallow: / User-agent: MojeekBot Disallow: / Section end What could be the problem here, then? Why does the Moz bot think I'm blocking it?
Link Explorer | | superseopl0