When I try creating a sitemap, it doesnt crawl my entire site.
-
We just launched a new Ruby app at (used to be a wordpress blog) -
We have not had time to create an auto-generated sitemap, so I went to a few different websites with free sitemap generation tools. Most of them index up to 100 or 500 URLS. Our site has over 1,000 individual listings and 3 landing pages, so when I put our URL into a sitemap creator, it should be finding all of these pages. However, that is not happening, only 4 pages seem to be seen by the crawlers.
TheSquareFoothttp://www.thesquarefoot.com/http://www.thesquarefoot.com/users/sign_inhttp://www.thesquarefoot.com/searchhttp://www.thesquarefoot.com/renters/sign_upThis worries me that when Google comes to crawl our site, these are the only pages it will see as well. Our robots.txt is blank, so there should be nothing stopping the crawlers from going through the entire site. Here is an example of one of the 1,000s of pages not being crawled****http://www.thesquarefoot.com/listings/Houston/TX/77098/Central_Houston/3910_Kirby_Dr/Suite_204Any help would be much appreciated!
-
Thanks for you help, can I ask one more question -
We just submitted a new sitemap to google for our new rails app -
http://www.thesquarefoot.com/sitemap.xml
Which has over 1,300 pages, however Google is only seeing 114. About 1,025 are in the listings folder / 250 blog posts / and 15 landing pages.
Any help would be appreciated!
Aron
-
I'd worry less about the sitemaps and more about internal linking structure. The problem you are having with crawlers is as symptom of the linking problem.
Most of your content seems to be on the other side of a search form. When crawlers, including those from search engines, explore you site they are looking for href links to follow - they will not submit forms.
If then you want the other content to be indexed then you need to provide a crawl path to it. Could you add links to each neighbourhood on page somewhere so that there is path to follow? That might lead on to further questions about your url structure and use of ajax too.
The general principal is that you should link to content you want to rank. Many will argue that a sitemap removes that necessity, but links provide more information that a list of URLs and I certainly wouldn't rely on sitemaps alone to get content indexed let alone ranked.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When the site's entire URL structure changed, should we update the inbound links built pointing to the old URLs?
We're changing our website's URL structures, this means all our site URLs will be changed. After this is done, do we need to update the old inbound external links to point to the new URLs? Yes the old URLs will be 301 redirected to the new URLs too. Many thanks!
Intermediate & Advanced SEO | | Jade1 -
Splitting One Site Into Two Sites Best Practices Needed
Okay, working with a large site that, for business reasons beyond organic search, wants to split an existing site in two. So, the old domain name stays and a new one is born with some of the content from the old site, along with some new content of its own. The general idea, for more than just search reasons, is that it makes both the old site and new sites more purely about their respective subject matter. The existing content on the old site that is becoming part of the new site will be 301'd to the new site's domain. So, the old site will have a lot of 301s and links to the new site. No links coming back from the new site to the old site anticipated at this time. Would like any and all insights into any potential pitfalls and best practices for this to come off as well as it can under the circumstances. For instance, should all those links from the old site to the new site be nofollowed, kind of like a non-editorial link to an affiliate or advertiser? Is there weirdness for Google in 301ing to a new domain from some, but not all, content of the old site. Would you individually submit requests to remove from index for the hundreds and hundreds of old site pages moving to the new site or just figure that the 301 will eventually take care of that? Is there substantial organic search risk of any kind to the old site, beyond the obvious of just not having those pages to produce any more? Anything else? Any ideas about how long the new site can expect to wander the wilderness of no organic search traffic? The old site has a 45 domain authority. Thanks!
Intermediate & Advanced SEO | | 945010 -
Where is the best place to put a sitemap for a site with local content?
I have a simple site that has cities as subdirectories (so URL is root/cityname). All of my content is localized for the city. My "root" page simply links to other cities. I very specifically want to rank for "topic" pages for each city and I'm trying to figure out where to put the sitemap so Google crawls everything most efficiently. I'm debating the following options, which one is better? Put the sitemap on the footer of "root" and link to all popular pages across cities. The advantage here is obviously that the links are one less click away from root. Put the sitemap on the footer of "city root" (e.g. root/cityname) and include all topics for that city. This is how Yelp does it. The advantage here is that the content is "localized" but the disadvantage is it's further away from the root. Put the sitemap on the footer of "city root" and include all topics across all cities. That way wherever Google comes into the site they'll be close to all topics I want to rank for. Thoughts? Thanks!
Intermediate & Advanced SEO | | jcgoodrich0 -
301 from one site to another
I have two e-commerce websites and i'm going to remove some products from website as requested by a supplier and sell them only on one site. Is it a good idea to 301 redirect the pages from site 1 to site 2?? Thanks for your help
Intermediate & Advanced SEO | | Aikijeff0 -
SEO for bigcommerce site
I have a site on bigcommerce platform .from Where do i need start SEO for these types of ecommerce sites.Looking for Experts ideas . Thank you.
Intermediate & Advanced SEO | | innofidelity0 -
Separate Site or should we incorporate it into our main site
Hello, We have a website to sell personal development trainings. The owners want to start 2 blogs - one for each owner - that promotes their personal coaching practices. What's the SEO advantages of embedding both blogs in the current site vs starting 2 brand new blogs with their names as the domain names?
Intermediate & Advanced SEO | | BobGW0 -
What is wrong with my site?
I could use suggestions/input. My site is consistently being beaten by doorway pages with zero to no content. In fact, on the keyword "lawrenceville plumber" I am being beaten by a KID with a blank wordpress installation. I really need help determining what my issues are and what I can do to help. According to all of the graders, ranks, etc my site is great. I have written tons of unique content, have added a blog unique articles and self-helps. Please help! My site is www.akinsplumbing.net. Thanks!
Intermediate & Advanced SEO | | chuckakins0 -
On-Site Optimization Tips for Job site?
I am working on a job site that only ranks well for the homepage with very low ranking internal pages. My job pages do not rank what so ever and are database driven and often times turn to 404 pages after the job has been filled. The job pages have to no content either. Anybody have any technical on-site recommendations for a job site I am working on especially regarding my internal pages? (Cross Country Allied.com)
Intermediate & Advanced SEO | | Melia0