Moz Crawl shows over 100 times more pages than my site has?
-
The latest crawl stats are attached. My site has just over 300 pages?
Wondering what I have done wrong?
-
total pages is higher you are right Keri but still only 581
-
I believe this image looks at what's indexed that's a subset of your sitemap that you submitted. You may want to look at Google Index -> Index Status in GWT to see what it shows there.
-
latest Moz crawl
-
latest webmaster tools crawl
-
I will definetly be paying attention to those numbers Keri. Webmaster tools is showing the right number of pages (something over 300 with 90% of those indexed)
-
It's not going to be a penalty, but it'll be good to have a bit less of a load on your server (bots no longer crawling thousands of pages) and just have your real pages in the index.
Places to look for interesting changes in site metrics would be your organic traffic in analytics and taking a look at your Google Webmaster Tools account to see your impressions, pages crawled, etc.
-
Thanks Keri, I will update asap.
could you let me know how big an issue would this be? (When you have the time of course;))
-
You're welcome! I may have opened a can of worms, however. That sitemap is generated by an automated tool (based on the footer at the bottom), so somehow it's finding that page 28 as well.
You may also want to ask the developer if you should be indexing the categories in the blog archives. There are resources on Moz about the best way to set that up in Wordpress, but I don't have them at my fingertips at the moment (I have a snuggly baby sleeping on my lap instead that's slowing me down a tad).
To answer your next question, after you figure out where the page 28 is being linked from and cure that, yes, you can do a one-time crawl from Research Tools. It won't overwrite your campaign info, but you can at least see if Moz is seeing thousands of pages or just a few hundred to see if stuff was fixed. Again, happy to provide more detail if/when you need it (and others will likely jump in with help on the thread, too).
I'd love to also see a little update a few weeks down the line of any changes you've noticed on your site metrics after getting this fixed.
-
You rock:)
-
And I found it. The sitemap at http://www.nineclouds.ca/sitemap includes a page /28, which is where the crawlers are finding the non-existent pages.
-
If you look at http://www.nineclouds.ca/blog/page/23, you'll see that there's a double arrow in the pagination at the right that goes to page 24, even though the last page is page 21. Google somehow has found the pages greater than 21 (which I'm not sure how they found), and once they found one of those, they keep seeing the link there with the double arrows to go to another page. Same happened with Rogerbot. I'm not sure where the bad originating link is (what legit page on your site is linking to something over page 21), but that's the loop that's happening and causing a ton of pages to be indexed. Get rid of those, and you'll also get rid of most of your errors.
-
Not shy about that at all thanks Keri.
any help you can provide is greatly appreciated.
-
Hi Bill,
Using my admin powers, I took a peek at your account. I'm still trying to figure out where it's coming from, but you have thousands of empty pages of your blog indexed. I'll dig around a little more and see if I can figure out what's up.
If you're comfortable with sharing your URL here in a public forum, other people can come take a look too. Otherwise, I'm happy to send you a private message with part of what's up and give your developer a place to start looking.
-
Thanks Keri. I am the owner of the site not the programmer so I am looking up the terms you are using as I write this response. If I am using pagination is there a way for the moz not to allow for this? If I understand your question about the calendar correctly I do have one as part of my blog that dates each post? Can I get the bot to not recognize this calendar?
-
My first guess would be parameters or something are being crawled. Do you have pagination? Sorting ascending and descending? A calendar that's getting crawled through the year 2525?
Your next step would be to look into what those duplicate pages are and see if something is amiss that's generating a ton of URLs.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google analytics - Users by time of day for speicific country?
Is there any way in analytics where we can check Users by time of day for speicific country?
Reporting & Analytics | | BPLLC0 -
Two long established sites with similar audiences, what do we do?
Hi guys, We operate two long established and reasonably well ranking sites — our company website which was built on a keyword domain: market-stalls.co.uk (approx 15 years online) and our online store which was established several years later on a different domain: tradersupplies.co.uk (approx 9 years online). (At the bottom of this post I've attached real world traffic and turnover figures that demonstrate the issue we're facing) The problem is... The above sites target very similar audiences and keywords and both rank fairly well but I know are likely competing against eachother We're a small company (8-10 employees) and we (or rather, I) don't have the time or resources to blog, build back links, manage opseo and all the social channels etc for both sites. I'm struggling to cope with one. The question is... Do we abandon the original company site (market-stalls.co.uk) in favour of pooling all our resource in to improving rankings for our online store (tradersupplies.co.uk). All our social media presence relates to tradersupplies.co.uk. We don't have any social channels for market-stalls.co.uk. Ironically, the only blog we have is established on market-stalls.co.uk — set up a couple of years ago in the hope to pull ourselves back up the rankings — but it hasn't been updated in over a year due to time restraints. Or do we attempt to keep both sites operational, despite a lack of resource? That would likely include a fairly sizeable overhaul of market-stalls.co.uk to bring it up to date with modern design standards, establishing social media channels for market-stalls.co.uk, creating a blog on tradersupplies.co.uk, and regularly updating two blogs and two sets of social media channels with unique content. Sounds like a pretty huge job right!? Obviously, had we been setting up our business in 2017 and having read the many community posts on the subject of multiple websites, we wouldn't be splitting our time between two websites and would be focussing solely on building one highly ranking site. But unfortunately we're not in this position and we're in a quandary because we don't know whether or not we should let our original, highly ranking company site drop off the radar in favour of focussing on building traffic to our online store. This situation arose out of a decision to establish our online store on a different domain to our company website. Back in 2007 I rebuilt market-stalls.co.uk and spent a lot of time optimising it. The site blew up and we were ranking very well for all kinds of keywords related to market stalls In 2009 we opened our online store tradersupplies.co.uk which sells all of the products advertised on market-stalls.co.uk and then some By using "buy now" buttons on market-stalls.co.uk which redirected to tradersupplies.co.uk, our original site was driving a large amount of traffic and sales to tradersupplies.co.uk. At it's peak it was driving almost £6,000 GBP a month in sales. This has since dropped to around a third/quarter of this total. As the business grew we began to run short of time to maintain market-stalls.co.uk and it has inevitably slipped down the rankings This has also had a direct impact on the referral traffic and resulting sales on tradersupplies.co.uk. I've attached below the analytics which show the drop in referral traffic to tradersupplies.co.uk and the drop off in sales. I have a feeling I know the answer to this debacle but I'm keen to hear the opinions of those that may have found themselves in this position before! UPDATE: I've just had a call with our Magento developer halfway through writing this post ... he has suggested we transfer all content from market-stalls.co.uk over to CMS pages on our Magento powered online store, and create 301 redirects. Apparently this will carry the weight of market-stalls.co.uk over to tradersupplies.co.uk. Does anyone have any thoughts on this? turnover.jpg
Reporting & Analytics | | tinselworm0 -
Webmaster Tools Suddenly Asking For Verification of Site Registered for 5 Years
Google Webmaster Tools has been successfully installed on my website, (www.nyc-officespace-leader.com) for more than five years. Suddenly, today I have received a request to Verify this Site". This makes no sense. The only possibility I can think of is that this is somehow tied to the following events in the last month: 1. Launch of new version of website on June 4th
Reporting & Analytics | | Kingalan1
2. Installation of Google of Tag Manager
3. Sudden Increase in number of pages indexed by Google. Unexplained indexing of an additional 175 pages. About 625 pages should be indexed, while 800 are now indexed. In the last month ranking and traffic have fallen sharply. Could it be tat these issues are all linked? But the strangest issue is the request to verify the site. Does anyone have any ideas? Thanks,
Alan0 -
Impressions in GWT have dropped to nothing, but my page is still ranking normally
Hello Everyone, I'm seeing a strange issue. On the 22nd of this month Webmasters tools started showing 6 impressions per day down from hundreds or thousands. I thought I was hit with a huge penalty for my keywords but they are still ranking where they have for the past month or two on Google. In analytics my organic traffic is stable. It just seems to be GWT showing the massive drop. My domain is: http://Patchofland.com Any Thoughts? Thanks in advance!
Reporting & Analytics | | PatchofLand0 -
Verifying Site Ownership & Setting Up Webmaster tools for clients who use Hubspot
We are a Hubspot partner agency. I'm trying to find the best route for managing Google's tools as an extra resource for insight, not the primary basis for marketing effort. I also want to explore adwords in more depth. Finding a lot of our clients don't have one or the other or both Analytics/Webmaster tools in place. Can I verify site ownership to set up webmaster tools simply by having admin access to their analytics account or will that require ownership of the analytics account? With Google merging things together these days I'm not sure of the best approach to take. Usually clients have their site hosted somewhere and built on some platform and ADD a Hubspot blog and the landing pages/cta's, Hubspot tools on a subdomain hosted by Hubspot. Hubspot has tools in it's website settings for adding google analytics (actually it's just a field to add code to the header area). If a client has universal analytics on their primary domain do I still need to go and add a separate analytics property for the subdomain and go through Hubspot's tools to install it on the subdomain? Or just use the same code from their primary domain and add it to the Hubspot header? What is the best route? Any additional thoughts on this subject are welcome - with so much updating and changing coming from Google (and Hubspot as we implement 3.0 - COS) I'm trying to avoid wasted effort, outdated methods, etc. Thanks!
Reporting & Analytics | | rhgraves651 -
Page Rank - logarithmic or exponential
Possibly a really stupid question. Is Page Rank logarithmic or exponential? I've seen a lot of people talking about Page Rank saying it's logarithmic but when they describe it they're actually talking about an exponential scale. (Apologies if I'm showing a basic misunderstanding in mathematical knowledge - I studied Drama)
Reporting & Analytics | | BenFox0 -
Stats show /blog/wp-cron.php at the top. What is it?
Hi, I have worked with websites for years but have no clue when it comes to Wordpress. We have our main website and then a Wordpress blog running in a subfolder that is only about a year old. The blog has only 7 posts so you can see how small it is vs main website with 200 pages. Usually our main index page of the site is at the top of the stats with the most views and this page /blog/wp-cron.php is about 30% lower. Now suddenly over the last month this page has jumped to the top and accessed almost as much as the home page of the site. We took a big hit with the latest Google Update so we are tyring to determine if there is anything technical in our site that has caused an issue. Thanks in advance Force7
Reporting & Analytics | | Force70 -
Google Analytics | REAL TIME
So I noticed today that there is now Real Time Data: http://analytics.blogspot.com/2011/09/whats-happening-on-your-site-right-now.html and I cannot figure out how to access this.
Reporting & Analytics | | joseph.chambers1