Way has the number of pages crawled plummeted?
-
Why has the number of pages crawled for our campaign plummeted in Moz Analytics – down to 729 from over 10k? Don't see any issues in Google Analytics with crawling our site.
-
Wow, that was like an hour of digging--trying to track down the linkscape crawler. I think I've got it now.
-
Rogerbot is the campaign crawler, and OSE has a different name. I know it can be a tad bit confusing at times.
-
Ah huh, I see. So does the campaign crawler have a different name or does it go by Rogerbot too?
-
Chris, what you're talking about is Open Site Explorer. I believe what Eyeglass Guy is talking about is the campaign crawl, which is an entirely separate crawl system.
-
Eyeglasses,
I wonder if your Mozrank or Moztrust has ratcheted down, as well, or if you've made any architectural/navigational changes to the site. With that many pages, it is possible that its not getting crawled as deeply as it had in the past if those dropped pages no longer met the value criteria for crawling by rogerbot--maybe you lost a good link or two or something like that. Also, if you changed any internal linking, it might be that those pages may have been pushed down in the architecture and not be as accessible as they once were.
-
How many pages does your site have? Did you possibly clear up a bunch of parameters and duplicate content, so the crawl is more accurate?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why can I see 404 pages in Google Analytics but nothing in the On-Demand Crawl?
Hello, I'm looking at some Google Analytics data for a website and can see a few 'Page not found's among the Page Titles, looking like these are 404 errors. To get a full list of what's 404-ing so I can get these redirected, the Moz on-demand crawl of the website has come back with no major errors and just a few metadata ones. Does anyone know any potential reasons why the audit has drawn a blank, and is there another way to get a comprehensive list of 404s, as I'm aware the Google Analytics data may not be covering all of them. Thanks very much Becky
Moz Bar | | becky.jenkins0 -
How do I see amount of traffic to a particular page on a website?
I am trying to figure out how much traffic goes to a particular page on a website. How would I go about finding this?
Moz Bar | | jbcorcoran6 -
Page Grader states "includes Canonical Tag" but it's not in the page source at all
I've ran it multiple times and changed other things it picked up on so not sure where it's getting the canonical tag is included even though it isn't?
Moz Bar | | Wana-Ryd0 -
On-Page Grader URL inaccessible when copy/pasted but not when edited
Hi!, I've looked through multiple topics on this but none quite seem to fit what's going on - hopefully someone can help! I get the error message 'Sorry, but that URL is inaccessible.' when I copy and paste a url from my site into the search e.g. http://www.orbussoftware.com/enterprise-architecture/ However if I edit this to https the search completes fine. Since we redesigned our site approx 6 months ago, we've found most of our rankings have completely dropped off, and now I'm getting this error I'm wondering if it has something to do with how our site is structured? If I'm getting this error with Moz does that mean Google could be having issues too? Or is it all just a strange quirk? Thanks!
Moz Bar | | JennaOrbus0 -
Https address has different result that http in Page Optimization Score toll in Moz PRO
The following url
Moz Bar | | TrueluxGroup
https://www.whichledlight.com/t/gu10-led-bulbs has (100 score for keyword 'GU10 LED')
has different on page opmisation score results to
http://www.whichledlight.com/t/gu10-led-bulbs (73 score for keyword 'GU10 LED') Anyone know if we've set something up wrong?
Also, is this even something to worry about, does google treat them differently? We're using the Page Optimisation Tool in Moz Pro ** UPDATE ** It's worth mentioning we are using emberjs, so the website is a single page application.
We use prerender to render the pages for google.0 -
How do I cancel a crawl request?
I was farting around and exploring the Crawl Test tool, and accidentally sent out a crawl for a competitor's site (I wanted to see if the tool would decline to crawl without verification). I do NOT want to actually crawl that site, nor do I want the competitor to see that we requested it (for obvious reasons) - how do I cancel it?
Moz Bar | | mkbeesto0 -
Odd crawl test issues
Hi all, first post, be gentle... Just signed up for moz with the hope that it, and the learning will help me improve my web traffic. Have managed to get a bit of woe already with one of the sites we have added to the tool. I cannot get the crawl test to do any actual crawling. Ive tried to add the domain three times now but the initial of a few pages (the auto one when you add a domain to pro) will not work for me. Instead of getting a list of problems with the site, i have a list of 18 pages where it says 'Error Code 902: Network Errors Prevented Crawler from Contacting Server'. Being a little puzzled by this, i checked the site myself...no problems. I asked several people in different locations (and countries) to have a go, and no problems for them either. I ran the same site through Raven Tool site auditor and got some results. it crawled a few thousand pages. I ran the site through screaming frog as google bot user agent, and again no issues. I just tried the fetch as Gbot in WMT and all was fine there. I'm very puzzled then as to why moz is having issues with the site but everyone is happy with it. I know the homepage takes 7 seconds to load - caching is off at the moment while we tweak the design - but all the other pages (according to SF) take average of 0.72 seconds to load. The site is a magento one so we have a lengthy robots.txt but that is not causing problems for any of the other services. The robots txt is below. Google Image Crawler Setup User-agent: Googlebot-Image
Moz Bar | | Arropa
Disallow: Crawlers Setup User-agent: * Directories Disallow: /ajax/
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
#Disallow: /js/
#Disallow: /lib/
Disallow: /magento/
#Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/
Disallow: /catalog/product
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/ Files Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt Paths (no clean URLs) #Disallow: /.js$
#Disallow: /.css$
Disallow: /.php$
Disallow: /?SID= Pagnation Disallow: /?dir=
Disallow: /&dir=
Disallow: /?mode=
Disallow: /&mode=
Disallow: /?order=
Disallow: /&order=
Disallow: /?p=
Disallow: /&p= If anyone has any suggestions then please i would welcome them, be it with the tool or my robots. As a side note, im aware that we are blocking the individual product pages. Too many products on the site at the moment (250k plus) which manufacturer default descriptions so we have blocked them and are working on getting the category pages and guides listed. In time we will rewrite the most popular products and unblock them as we go Many thanks Carl0 -
Is there a way to get Page Authority values included in the Crawl Diagnostic .csv export?
Would be nice to have these values included so that you can sort by Page Authority. 4uF6efx.png
Moz Bar | | WebReputationBuilders0