Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email [email protected] about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email [email protected] they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
URL for a new website
Hi, I am creating a new website for a client. Is it best to include the keywords from the most common search in the domain name, they would like: forenamesurname.com but should I be recommending: weddingmakeupbyforename.com Does it make much difference to search rankings if the keyword is in the domain name? Thanks v much
On-Page Optimization | | danieldunn100 -
Page rank question
Hi all, pretty standard SEO question that I'm sure you are all getting asked day in, day out... the following page: http://www.3wisemonkeys.co.uk/dvd/portable-dvd-player-car/ It's targetting keywords "portable dvd", "portable DVD player", "portable DVD players" & "portable dvd player for car". The page gets a grade A via Moz analytics On-page grader for the keywords. We have incoming links and I believe the page is structured correctly with all meta tags, H1 etc and tried not too overload the tags with keywords. We have the keywords in the URL etc. We are still stuck on page 2 for all phrases. Any thoughts or suggestions as to what we are doing wrong please? Any help would be great.
On-Page Optimization | | jasef0 -
Page Analysis - Helping Product Pages Outrank Search Results Pages
Hi! We have a lot of our search results pages that have been indexed and outrank our product pages and in some instance the actual product pages barely show up at all. Here is an example query that includes our brand name: http://goo.gl/cgB6W So, we have loads of actual product pages, video pages, etc that should be showing up here, but are not and this is just one example. Unfortunately, there are a LOT of these Search Results pages out there and utlimately we would love to de-index them altogether, but it's going to have to be carefully done. So, was wondering if anyone would want to check out one of our product pages and give any feedback as to what we could change to possibly improve rank or to make them more search friendly or hopefully to help them rise above these indexed search results pages? Here is an example product page: http://goo.gl/2R4IT Thanks!! Craig
On-Page Optimization | | TheCraig0 -
How to schedule the on page reports myself
The on page reports are scheduled on mondays, but is there a way to schedule it my self.
On-Page Optimization | | JoostBruining0 -
On Page Optimization Report
Does this tool also guard against an instance of over-optimization or keyword-spamming?
On-Page Optimization | | webfeatus0 -
Page title changes based on results per page
I have a product listing that allows customers to set a results per page option. This ads a GET variable to the URL. I've added the page number to the title of this page if they go past the first page. If this results per page variable is added to the URL then google see it as a different page. Do I need to change the page title for this?
On-Page Optimization | | BedInABox.com0 -
I am optimizing my webpages according to suggestions from the On Page Report Card. Should I have more than one keyword for a page?
I am optimizing my webpages according to suggestions from the On Page Report Card. Should I have more than one keyword for a page or should I make separate pages for each keyword even when they are similar? Will Google penalize me for making similar pages? Imagine selling, bargain milk chocolate peanut clusters. Keywords examples could be: Bargain chocolate Bargain milk chocolate Bargain milk chocolate peanut clusters Bargain chocolate peanut clusters Chocolate peanut cluster bargains Milk chocolate peanut cluster bargains Etc. Will one page called http://mycompany/bargainmilkchocolatepeanutclusters.com be OK or should I have one called http://mycompany/bargainmilkchocolate.com and one called http://mycompany/bargainmilkchocolatepeanutclusters.com and one called http://mycompany/chocolatepeanutclusterbargains.com , etc.? Thanks for your advice.
On-Page Optimization | | KSHAYY0 -
SEO for One Page Websites
Hi Are there any SEO guidelines for "one page websites". I'm looking into the 'benefit' it might have in combination with exact match URLs. Many thanks in advance.
On-Page Optimization | | Partouter2