Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email [email protected] about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email [email protected] they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Page Titles
It seems as though we are being flagged for duplicate page titles when really they are slightly different. Is it better to remove the "dart board" or "dart board backboard" from all the product titles? We were doing this for optimal SEO - to rank for the search of "dart board" - but is it really hurting us? for example, our product titles are: Obama dart board backboard, Texas dart board backboard, Oklahoma dart board backboard, etc. Yet they are being flagged as duplicate titles.
On-Page Optimization | | DartsDecor0 -
Finacial pages markup
I was wondering if there is away to use Schema for stock market pages?
On-Page Optimization | | ciznerguy0 -
The seomoz on page keyword analysis tool is not showing title or keyword in document
the SEOMOZ onpage analysis tool is not not showing title or keyword for any page in one of my sites. It says there are no title elements on my page and there are, i checked the source code myself and they are there and correct. my title and keywords are in there and show up fine in firefox and internet explorer even after i refresh them. why would this tool show them as missing in one of my sites but not others? I'm worried that google's spider might not see them if the on page analyzer doesn't see them and my rankings might drop. they showed up the other day in the seomoz on page analyzer just fine and i haven't changed anything. Thanks mozzers!
On-Page Optimization | | Ron100 -
PAGE TİTLE
<title> </span>Home to home moving 4356 <span></title> page A <title> </span>Home to home moving 3723 <span></title> page B These two titles are the same?
On-Page Optimization | | iskq0 -
Sold Property Page
I have a real estate agent property website. What should I do with the sold property page. Should I remove the page entirely since the page has no more use for user. Currently the page is still exist, but only with word "This property is currently not available/sold"
On-Page Optimization | | odihost10 -
Why is SEOMoz telling me that meta descriptions are missing?
We have descriptions using an Ektron CMS on a .NET platform. IT is wondering if using "meta ID" vs "meta name" is the issue? Here is an example of the HTML code: <meta http-equiv="<a class="attribute-value">Content-Language</a>" name="<a class="attribute-value">Language</a>" content="<a class="attribute-value">en</a>" /> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaKeywords</a>" name="<a class="attribute-value">Keywords</a>" content="<a class="attribute-value">hunting, fishing, outdoor, sportshooting, rodeo, off-road, hunt, fish, outdoors, community, shotgun, rifle, bass, trout, sportfishing, tarpon, ATV, television</a>">meta> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaDesc</a>" name="<a class="attribute-value">Description</a>" content="<a class="attribute-value">Deep-cranking technique lands Starks his second Elite Series victory.</a>">meta> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaSubject</a>" content="" name="<a class="attribute-value">subject</a>">meta> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaRating</a>" content="" name="<a class="attribute-value">rating</a>">meta>
On-Page Optimization | | cmoise0 -
Blog page outranks static page for KW -- why?
Blog page ranks 10 in Google, while the static page is on page 7. What makes it more interesting is that the blog page scores an "F" with the Term Target tool while the static page scores an "A". Static page has more inbound links and a mR/mT of 3.89/ 4.54 vs. 3.71/ 4.14 for the blog page. Any ideas on how to approach this one?
On-Page Optimization | | 540SEO0 -
Why some pages are not indexed?
I have a furniture´s ecommerce. When searching for "site: movstore.com.br" returned 1080 results, but if I search for "site: movstore.com.br / Product" returned 1020 results. I mean, that 1080 indexed pages, 1020 are products pages and the other 60 pages are irrelevant. Where are the category pages? "site: movstore.com.br / Categories" - 0 results
On-Page Optimization | | maisempresas
"site: movstore.com.br / Departments" - 0 results
"site: movstore.com.br / Marks" - 0 results What might be happening?0