When I try creating a sitemap, it doesnt crawl my entire site.
-
We just launched a new Ruby app at (used to be a wordpress blog) -
We have not had time to create an auto-generated sitemap, so I went to a few different websites with free sitemap generation tools. Most of them index up to 100 or 500 URLS. Our site has over 1,000 individual listings and 3 landing pages, so when I put our URL into a sitemap creator, it should be finding all of these pages. However, that is not happening, only 4 pages seem to be seen by the crawlers.
TheSquareFoothttp://www.thesquarefoot.com/http://www.thesquarefoot.com/users/sign_inhttp://www.thesquarefoot.com/searchhttp://www.thesquarefoot.com/renters/sign_upThis worries me that when Google comes to crawl our site, these are the only pages it will see as well. Our robots.txt is blank, so there should be nothing stopping the crawlers from going through the entire site. Here is an example of one of the 1,000s of pages not being crawled****http://www.thesquarefoot.com/listings/Houston/TX/77098/Central_Houston/3910_Kirby_Dr/Suite_204Any help would be much appreciated!
-
Thanks for you help, can I ask one more question -
We just submitted a new sitemap to google for our new rails app -
http://www.thesquarefoot.com/sitemap.xml
Which has over 1,300 pages, however Google is only seeing 114. About 1,025 are in the listings folder / 250 blog posts / and 15 landing pages.
Any help would be appreciated!
Aron
-
I'd worry less about the sitemaps and more about internal linking structure. The problem you are having with crawlers is as symptom of the linking problem.
Most of your content seems to be on the other side of a search form. When crawlers, including those from search engines, explore you site they are looking for href links to follow - they will not submit forms.
If then you want the other content to be indexed then you need to provide a crawl path to it. Could you add links to each neighbourhood on page somewhere so that there is path to follow? That might lead on to further questions about your url structure and use of ajax too.
The general principal is that you should link to content you want to rank. Many will argue that a sitemap removes that necessity, but links provide more information that a list of URLs and I certainly wouldn't rely on sitemaps alone to get content indexed let alone ranked.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Updating 2013 Site Built with Custom Theme, Modify Existing Theme, Create New Custom Theme, Or Use Child Theme?
Our website was designed in 2013 using a custom theme. Some of the plugins are built from scratch. Ranking in our industry is hyper competitive. We are seeking a better interface and also to improve ranking. I have read that custom themes use lighter code and can rank better. Does this apply to a custom theme from 2013? Will we have an SEO advantage using a custom theme? If so, will that advantage be significant? We are using a discontinued plugin called "Firestorm" to display real estate listings. That plugin has been customized. Can we use that plugin on a new "custom" theme? How about on a "child" theme? In terms of the cost of future maintenance, will a "custom" theme require much more intervention (manual installation of updates) moving forward? Which of the following options is best: 1. Adapt our existing custom theme
Intermediate & Advanced SEO | | Kingalan1
2. Create a new custom theme
3. Create a child theme Thanks,
Alan0 -
My site shows 503 error to Google bot, but can see the site fine. Not indexing in Google. Help
Hi, This site is not indexed on Google at all. http://www.thethreehorseshoespub.co.uk Looking into it, it seems to be giving a 503 error to the google bot. I can see the site I have checked source code Checked robots Did have a sitemap param. but removed it for testing GWMT is showing 'unreachable' if I submit a site map or fetch Any ideas on how to remove this error? Many thanks in advance
Intermediate & Advanced SEO | | SolveWebMedia0 -
1 Ecommerce site for several product segments or 1 Ecommerce site for each product segment ?
I am currently struggling with the decision whether to create individual ecommerce sites for each of 3 consumer product segments or rather to integrate them all under one umbrella domain. Obviously integration under 1 domain makes link building easier, but I am not sure how far google will favor in rankings websites focussed on one topic=product segment. Product segments are medium competitive.Product segments are not directly related but there may be some overlap in customer demographics- Any thoughts ?
Intermediate & Advanced SEO | | lcourse1 -
Creating 20+ websites with links back to central site
Hey guys, A client of ours owns an IT company with 20+ locations across the UK. He is looking for a solution to provide each of their 20+ locations with a page or website that they can manage themselves that links directly back to the main site. His idea is to create 20+ one or two page websites that could all link back to the main central site - aiding the possibility of ranking well for locally-based terms. At the moment, we have a page for each of the 20+ locations on the main site. However, the client wants to give his franchisees complete control over their web presence. Would a setup like this work? Would it be logical to have 20+ websites (likely to follow a very similar format) all pointing to one central website? Would we have to "no-follow" links back to main site in order to show we aren't trying to manipulate page rank? Would creating sub folders on the main site be a better option for each of the 20+ locations? Any feedback appreciated!
Intermediate & Advanced SEO | | Webrevolve0 -
Create a link or redirect?
We have 60 demo movie pages on our site. We no longer link to these movie pages internally, because they are outdated; however, a lot of our partner companies are still linking to these pages. Some of these pages will have 10-15 linking root domains and a page authority of 30+... so pretty decent authority. These pages only include a movie on the pages, no links. I am trying to pass some of the link juice from these pages to other pages on our site. I am wondering if I should: A)Include transcripts on these pages, then link back to our current product page or solution pages? B)Set up redirects from these pages to a product or solution page? C)Set up a redirect to our homepage? Any advice? Thanks, Mike
Intermediate & Advanced SEO | | Mike.Goracke0 -
Why is SEO Moz only crawling my homepage?
When I send the SEO Bot to crawl my website, it only crawls the homepage? Does anyone know why this is happening? Thanks, Andy
Intermediate & Advanced SEO | | ChrisCurdDesign0 -
Sitemaps and subdomains
At the beginning of our life-cycle, we were just a wordpress blog. However, we just launched a product created in Ruby. Because we did not have time to put together an open source Ruby CMS platform, we left the blog in wordpress and app in rails. Thus our web app is at http://www.thesquarefoot.com and our blog is at http://blog.thesquarefoot.com. We did re-directs such that if the URL does not exist at www.thesquarefoot.com it automatically forwards to blog.thesquarefoot.com. What is the best way to handle sitemaps? Create one for blog.thesquarefoot.com and for http://www.thesquarefoot.com and submit them separately? We had landing pages like http://www.thesquarefoot.com/houston in wordpress, which ranked well for Find Houston commercial real estate, which have been replaced with a landing page in Ruby, so that URL works well. The url that was ranking well for this word is now at blog.thesquarefoot.com/houston/? Should i delete this page? I am worried if i do, we will lose ranking, since that was the actual page ranking, not the new one. Until we are able to create an open source Ruby CMS and move everything over to a sub-directory and have everything live in one place, I would love any advice on how to mitigate damage and not confuse Google. Thanks
Intermediate & Advanced SEO | | TheSquareFoot0 -
Building a mobile site.
We are building a mobile site that will be launching in another month. I’m concerned that the mobile site will start catabolizing our traditional rankings. Is there a way to keep this from happening? Should we utilize the cross domain canonical tag and point back to the traditional site URLs?
Intermediate & Advanced SEO | | SEO-Team0