Crawl test. Bot crawled only 200 or so links when it should have crawled thousands
-
Hi everyone,
I just recieved my crawl test report and its only given me 200 or so URL's when my site has thousands, any thoughts?
-
Hi Ryan,
I am the site owner and this is the precise reason im trying to take matters into my own hands.
<meta name="keywords" content="E60,Rear,lamp,set" /> I see what you mean, because this is actually ridiculous, not quite sure how it got into this state either. Whats that saying, "if you want something done you have to do it yourself" Looks like i have to take a crash course in SEO to sort it all out. Thanks very much for all your help.
-
I realize you may not have full control over the site. What I would share is:
"That's how the site is" is not an acceptable response, unless the site owner is satisfied with their current SEO ranking.
The keywords have NOTHING to do with the product being displayed on the page. The link I offered is for a Hella e60 Rear Lamp. The only related in the keyword section is "rear". I am quite certain that is by coincidence.
Your keywords are not dynamically generated to vary with the pages content, nor were they manually altered to fit the pages content. The keyword selection is awful. The numbers "3", "5", and "7" are listed as 3 of the key words.
I want to help you, so don't take this the wrong way. The best thing about that site is it probably qualifies as a textbook case of what NOT to do from a SEO perspective. Perhaps you can appeal to a SEO company to use the site in a case study and turn it around.
-
Thank you very much Ryan, the columns on two sides of the page cant be helped as thats, how the site is, only the central content changes. However the duplicate keywords are for the products themselves, for example i sell 50 different BMW oil filters. Theres not much i can do about duplicating keywords as all of the products are very very similar.
I think you might be right about the site redesign....
-
A few notes about your site:
-
you are using meta keyords in your header. It offers no benefit and I would suggest removing it. It's not related to your inquiry but is something I noticed.
-
your site has a 50 keyword TAG block with the same keywords on every page. This isn't good from a SEO perspective on many levels. You want your keywords to focus the unique content on a given page
-you site pages are likely viewed as all duplicates. I can recognize the main item in the center of the page changes, but would a crawler? Your left and right sidebars are identical on all pages, along with most of your header. The actual content you offer is only a small percentage of the total page.
The large image of the various car parts is not considered as part of the content, aside from the ALT tag.
Look at a random page from your site: http://www.incarmotorfactors.co.uk/content/16-hella-bmw-e60-rear-lamp-set
According to the Analyze tool there are 5975 words on the page. I estimate about 100 of them are unique words addressing your Rear Lamp product, and the remaining 5800+ words are exactly the same as every other product page.
A crawler will see your pages as 98% duplicated data and the result will likely be your site isn't going to be listed. I would recommend a site re-design. Before taking that advice, it would probably be best to hear from others who have a lot more SEO expertise then myself.
-
-
-
What is different about your site? Is it flash or javascript based? Can you share your site URL?
-
Hi Ryan,
I used
On-Page Optimization Tools: Crawl Test. However this problem may be deeper than i first thought, as SEOMoz is not able to read any of my site info properly.
Open site explorer cant read it
Linkscape cant read the links. Crawl test isnt read properly, however my server and robots.txt files are fine, theres no blocking attempts from the server. Very strange.
-
What Crawl Test tool did you use?
Depending on the crawl tool, it may not look at content blocked from your Robots.txt file. You may want to ensure it is configured correctly.
Are there any permission issues? The crawler will look at your site the way a guest would. Any content which requires users to log in would be hidden to the crawler.
Are there any other issues regarding your site's accessibility? Connection or firewall issues? Could a server admin have seen a server performance issue and kicked the crawler before it finished? You can check server logs for this information.
If you check everything and do not locate a definitive cause, I would suggest running the crawl once more and checking the results before pursuing the matter further.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Spam links with high DA
I'm running through the spam analysis and noticed a few links that are considered to be 9+ on the spam score (high spam level), but some of them have DA of 60 -75. How can this be possible? When I view the sites they appear to be be pretty low quality. If i'm considering disavowing a site should I focus on spam score or DA? A DA of 75 is a pretty reputable standing.
Moz Pro | | STP_SEO0 -
What impact has external links reported as http503
Hello I have 72 external broken links who are reported with a http 503 status. When I ask the owner of that site, he confirmed he has removed it from GA by using this status. My question is. Does this have an impact for the quality of my site? The site is available, there is no delay when you open the link. Thank you for your support. br
Moz Pro | | kjerstibakke
Kjersti Bakke0 -
Total External Links
Hi There, I'm looking at my link profile and my Total External Links dropped uber dramatically back in April. Any way to figure out what the heck happened? My only idea is that this was around the time we launched a new version of the website but I'm not sure it corresponds exactly and anyway the domain didn't change. Possibly some urls did but all have been redirected. Any ideas? Thanks, site is www.nationalpardon.org
Moz Pro | | NationalPardon0 -
Crawl Diagnostics : Problem of display in Excell.
Hi Mozers, I've just finished watching the Crawl Diagnostics Webinar and when I try to export one of my campaign into the CSV format, I've a display problem into Microsoft Excell. Every headtitles are into the "A" column so, I can't do anything with that : I can't organize the data,... It's totally unreadable. What can I do? Thank you for yours answers. Jonathan
Moz Pro | | JonathanLeplang0 -
Why do links appear then disappear month to month?
What I mean by this is, a link to my site shows up in the August mozscape crawl, then the next month it doesn't, but its still an active link on the website. Once a site is crawled, shouldn't it always be there from now on?
Moz Pro | | SeanConroy0 -
Crawl Diagnostics - unexpected results
I received my first Crawl Diagnostics report last night on my dynamic ecommerce site. It showed errors on generated URLs which simply are not produced anywhere when running on my live site. Only when running on my local development server. It appears that the Crawler doesn't think that it's running on the live site. For example http://www.nordichouse.co.uk/candlestick-centrepiece-p-1140.html will go to a Product Not Found page, and therefore Duplicate Content errors are produced. Running http://www.nhlocal.co.uk/candlestick-centrepiece-p-1140.html produces the correct product page and not a Product Not Found page Any thoughts?
Moz Pro | | nordichouse0 -
A question about Mozbot and a recent crawl on our website.
Hi All, Rogerbot has been reporting errors on our website's for over a year now, and we correct the issues as soon as they are reported. However I have 2 questions regarding the recent crawl report we got on the 8th. 1.) Pages with a "no-index" tag are being crawled by roger and are being reported as duplicate page content errors. I can ignore these as google doesnt see these pages, but surely roger should ignore pages with "no-index" instructions as well? Also, these errors wont go away in our campaign until Roger ignores the URL's. 2.) What bugs me most is that resource pages that have been around for about 6 months have only just been reported as being duplicate content. Our weekly crawls have never picked up these resources pages as being a problem, why now all of a sudden? (Makes me wonder how extensive each crawl is?) Anyone else had a similar problem? Regards GREG
Moz Pro | | AndreVanKets0 -
MozRank per Link: Valuable or Worthless?
Since the MozRank and MozTrust metrics are on a logarithmic scale, I'm wondering if it's still valuable to get a general idea of the quality of a site's link profile by calculating: MozRank / Total Links MozTrust / Total Links Would this be useful, or is it always going to be higher for sites with lower MozRank & MozTrust since it's "easier" to achieve the lower values?
Moz Pro | | conquerapathy-644120