Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email [email protected] about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email [email protected] they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My site on desktop browser: page 2 /mobile browser: page 0
Using my two most pertinent keywords in Chome my site shows up page two. Using the same keywords on my iPhone does not show my site at all (I clicked on to page 15). I have a mobile ranking of 84 on Google PageSpeed Insights. Could be a bit higher but not enough to totally ignore my site. What am I missing?
On-Page Optimization | | artsp0 -
Area pages
As area pages are seen as trying to game google (see link below) is their a 'better way' to target multipe areas (100 odd)? https://support.google.com/webmasters/answer/2721311?hl=en Cheers
On-Page Optimization | | webguru20140 -
Question About On Page Grades
New to the whole SEO game in some aspects and wanted to know about on page grades. Some of the pages I got ranks of F for are actually in the top 10 in Google. Could anyone help me to explain way these pages/tags are F's with those rankings?
On-Page Optimization | | BlackEnterprise0 -
What kind of pages are they?
Hi all, after making an analysis of my website the tool has found two pages that I don't know what refers to: /?page id=2058 (only one page of this kind) /?attachment id= (only one page of this kind) does anyone know what kind of pages they could be? Do the have relevancy regading SEO? The plattform is Wordpress. Thanks!
On-Page Optimization | | juanmiguelcr0 -
Too Many On-Page Links
Hi, I did a SEOmoz campaign and got results today, One of the results is Too "Many On-Page Links" when i am drilling down, i see that that's include inside links. for example, i sale food, i have my main department window - inside i have 30 products - each product is linked to a detailed page about the product. so automatically i have 30 links - not including all the others in this page, and i easily get over 100 and even sometimes 200 is this a big issue? does it damages my SEO? If yes, is there a way to write the HTML in a way that internal links like that wont be counted? Thank you SEOWiseUs
On-Page Optimization | | iivgi0 -
One Page Website vs. Multipage Site, if you want to target one specific Keyword only.
Hello! suppose I want to start a website about, let's say spray adhesives. My aim is to rank on the first page for the keyword "spray adhesive". I don't care about my ranking on more specific keywords like "Tesa spray adhesive" or "3M spray adhesive". My ranking for more general keywords like "glue" is unimportant, too. So I thought about creating a single-page website, that writes about spray adhesives, the pros & cons of every manufacturer, and shows the best discounts for spray adhesives. Each section can be accessed through a top-navigation, that links via anchors to the individual sections. The page will be updated every day On the other hand, i could create a blog and write an article for every specific spray adhesive. So I would have a home page that lists the latest articles for every product, with titles like "3M spray adhesive CreativeMount", "3M spray adhesive SprayMount", "Tesa Spray adhesive" ... I will write one article every day What do you think would be the better strategy? Is there a risk to create competing articles for the keyword "spray adhesive" and thus rank lower if I go with the blog strategy? On the other hand, does google rate singe-page websites lower, because google thinks those websites are less valuable than websites with many pages for the same topic? Thank you ver much for you help in advance!
On-Page Optimization | | MGMT0 -
On Page Optimisation Reports
Firstly sorry if this has already been answered - I did look I promise.
On-Page Optimization | | Jock
Secondly sorry if the answer to this is blatently obvious! In the process of trying to optimise my landing pages, I am using On Page Optimisation reports. I have several (ok lots) with F grades which is not surprising as the landing page is not the landing page optimised for a certain keyword. If I change the landing page to the one that I have for a certain keyword then hey presto A or B grade (clever me)! Now here's the thing - presumably the landing page that is listed by default is the one that Google "sees" for a particular keyword. How do I change this if I can or do I have to be patient or am I just being plain daft?! Many thanks0 -
SEOmoz crawl error
Hi, I'm getting a crawl error and it complains about there being missing meta description... But, the errors are all for non existent index files in directories that only contain pdf files and some thumbs of the front page... Just started trying to learn this stuff...! Cheers Rod
On-Page Optimization | | DrWho0