Can Google see all the pages that an seomoz crawl picks up?
-
Hi there
My client's site is showing around 90 pages indexed in Google. The seomoz crawl is returning 1934 pages.
Many of the pages in the crawl are duplicates, but there are also pages which are behind the user login.
Is it theoretically correct to say that if a seomoz crawl finds all the pages, then Google has the potential to as well, even if they choose not to index?
Or would Google not see the pages behind the login? And how come seomoz can see the pages?
Many thanks in anticipation!
Wendy
-
Well, that could be your easy solution. Make sure they're all set not to be indexed, then you'll be able to (mostly) ensure Google won't crawl them, and they'll probably disappear from your moz crawl report as well. As far has how moz is finding them to begin with behind your login wall, sorry, I have no idea.
-
The pages behind the login? No not yet - they are a new client, so I am just auditing at the moment to identify what we need to do
Many thanks for your replies!
-
This may be an obvious question, but to you have those pages set to noindex?
-
Hi Marisa
seomoz are crawling unecessary pages, (they return pages ignored by screaming frog for example)
BUT my concern is that if Google can also see them, even if they choose to ignore them my client maybe getting slammed for duplicate issues or the pages behind the login may suddenly appear in the index.
We'll get no index / no follow added, and fix the dupes, but am really interested as to how seomoz sees behind the login
-
Here's the real question: Do you WANT Google to see all these pages, or is SEOmoz crawling unnecessary pages?
-
Great, many thanks Nakul - they are a new client so am waiting on getting access to WMT - will go through with a fine tooth comb! Just seems really weird with regards to the pages behind the login ...
-
Wendy, if SEOMOZ can see it, I am sure Google can see it as well. I would login to your webmaster console and check the index status. Do you have an XML sitemap submitted for your website ? Once you do, you'll have a more accurate read on the number of pages you submitted and how many of them are indexed. The new index status Google introduced last month also lets you see pages Google ignored for multiple reasons.
I hope this helps.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pages with Temporary Redirects on pages that don't exist!
Hi There Another obvious question to some I hope. I ran my first report using the Moz crawler and I have a bunch of pages with temporary redirects as a medium level issue showing up. Trouble is the pages don't exist so they are being redirected to my custom 404 page. So for example I have a URL in the report being called up from lord only knows where!: www.domain.com/pdf/home.aspx This doesn't exist, I have only 1 home.aspx page and it's in the root directory! but it is giving a temp redirect to my 404 page as I would expect but that then leads to a MOZ error as outlined. So basically you could randomize any url up and it would give this error so I am trying to work out how I deal with it before Google starts to notice or before a competitor starts to throw all kinds at my site generating these errors. Any steering on this would be much appreciated!
Moz Pro | | Raptor-crew0 -
How can I deal with tag page duplicate issues
The Moz crawler reported some dupliated issues. Many of them have to do with tags.
Moz Pro | | IamKovacs
Each tag has a link, and as some articles are under several tags, these come up as duplicate content. I read Dr Peter's piece on Canonical stuff, but it's not clear to me if any of these are the solution. Perhaps the solution lies somewhere else? Maybe I need to block the robots from these urls (But that seems counter-SEO-productive) Thanks
Kovacs0 -
Aren't domain.com/page and domain.com/page/ the same thing?
Hi All, A recent Moz scan has turned up quite a few duplicate content notifications, all of which have the same issue. For instance: domain.com/page and domain.com/page/ are listed as duplicates, but I was under the impression that these pages would, in fact, be the same page. Is this even something to bother fixing or a fluke scan? If I should fix it does anyone know of an .htaccess modification that might be used? Thanks!
Moz Pro | | G2W0 -
Is there a report I can run to get a list of all pages indexed by Google for my website?
I want to get a CSV file of all the pages that are indexed by Google and other search engines so I can create and .htaccess file of 301 redirects
Moz Pro | | etraction0 -
How can competition outrank you if your site has better Domain/Page Authority, More links, and More Social sharing?
Say you have a site that has better Domain/page authority, more links, more social media sharing, and a lot more indexed pages (thanks to blogging) than the competition. Of course all of these metrics are based off of data from SEOMoz open site explorer tool which I am not sure if it produces accurate data. 1. Other than exact match domains or the age of a domain what would be other reasons why competition would outrank you? 2. Can anyone suggest other ways to help increase a sites domain/page authority besides creating more indexed pages, link building, etc..?
Moz Pro | | webestate0 -
Why can't I add my facebook page to SEOMOZ? Also having other facebook issues.
Hi, I have no trouble adding my twitter page in SEOMOZ, but its giving me an error when I try to load my facebook page http://www.facebook.com/pages/Eugene-Computer-Geeks/226660334011653 . I also tried adding my personal facebook page which is tied to the Eugene Computer Geeks facebook page, but SEOMOZ wont accept that either. My business facebook page is tied to my personal account, and its also not showing up on the facebook search. Any idea how I can make my business show up? I wish I could just start over fresh and have my buinsess setup with it's own facebook account. Thanks.
Moz Pro | | eugenecomputergeeks1 -
Does the SEOMoz weekly crawl that highlights no meta description tag, take into account if there is a meta robots noindex,follow tag on the pages it indicates the missing meta descriptions?
The weekly crawl website report is telling me that there are pages that have missing meta description tags, yet I've implemented meta robots tags to 'noindex, follow' those pages which are visible in those page source files. As far as Google Is concerned, surely this then won't be a problem since it is being instructed NOT to consider these specific pages for indexing. I am assuming that the weekly SEOmoz website crawl is simply throwing the missing meta description crawl findings into its report without itself observing that the particluar URL references contain the meta robots 'noindex,follow' tag ???? Appreciate if you can clairfy if this is the case. It would help me understand that (at least in terms of my efforts towards Google) your own crawl doesn't observe the meta robots tag instruction, hence the resultant report's flagging the discrepancy.
Moz Pro | | callassist0 -
What causes Crawl Diagnostics Processing Errors in seomoz campaign?
I'm getting the following error when seomoz tries to spider my site: First Crawl in Progress! Processing Issues for 671 pages Started: Apr. 23rd, 2011 Here is the robots.txt data from the site: Disallow ALL BOTS for image directories and JPEG files. User-agent: * Disallow: /stats/ Disallow: /images/ Disallow: /newspictures/ Disallow: /pdfs/ Disallow: /propbig/ Disallow: /propsmall/ Disallow: /*.jpg$ Any ideas on how to get around this would be appreciated 🙂
Moz Pro | | cmaddison0