My site is not being regularly crawled?
-
My site used to be crawled regularly, but not anymore. My pages aren't showing up in the index months after they've been up. I've added them to the sitemap and everything. I now have to submit them through webmaster tools to get them to index. And then they don't really rank?
Before you go spouting off the standard SEO resolutions...
- Yes, I checked for crawl errors on Google Webmaster and no, there aren't any issues
- No, the pages are not noindex. These pages are index,follow
- No, the pages are not canonical
- No, the robots.txt does not block any of these pages
- No, there is nothing funky going on in my .htaccess. The pages load fine
- No, I don't have any URL parameters set
What else would be interfereing?
Here is one of the URLs that wasn't crawled for over a month: http://www.howlatthemoon.com/locations/location-st-louis
-
Google may have reduced their crawl budget for your site if they found enough pages to be low quality, thin, duplicate, etc... Here are some examples that you can probably apply a noindex tag to in order to reduce crawl budget waste on them.
Google these:
site:howlatthemoon.com/ inurl:tag
1,360 indexed pages that don't need to be in the SERPssite:howlatthemoon.com/ inurl:tag page
Even if you want the tag pages indexed, you don't need their paginated pages indexed toosite:howlatthemoon.com/ inurl:page inurl:category
A few dozen category pagination pages in the SERPs, many more on the site that have been booted from the index
Example:
http://www.howlatthemoon.com/dueling_piano_bar/category/nightlife/page/2/I would install the Yoast Wordpress SEO plugin, which should fix these in the following ways:
Use rel canonical tags
Use rel Next/Prev tagsNoindex tag pages
etc... -
That page is indexed. Can you give me an example of a page that hasn't been indexed yet? Your onsite SEO seems fine except you should get rid of your meta keywords tag and your Meta Description tag is a bit long.
-
oh sorry! Here is one of the pages that wasn't crawled since it was posted in early November.
-
You forgot one...what is the URL of your site
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Mobile site not ranking
Hello, I have a m.site.com version of my original site. It is about 1/10 the size, and no matter what I do-I can't get the site to rank. I've added more pages and specified canonical etc etc. Should I add as many pages as my larger site has? Are there specific places I should be submitting this version beyond the typical? I am at a loss, so any help would be greatly appreciated! Thanks! L
Technical SEO | | lfrazer1 -
Open Site Explorer Question
In OSE I have 3 of my top 5 pages as store.com, store.com/Default.asp, and store.com/default.asp -- I have a canonical version of at store.com/default.asp. I have inbound links coming to all three urls -- b/c OSE is listing these as seperate pages does that mean the link juice is not being consolidated? Or is this not something to worry about?
Technical SEO | | IOSC0 -
Why is my crawl taking so long?
Hi There, My crawl for albertcuyp.nl is taking very long, it started on the 10th of april. I don't know whats going on but i think 2 weeks for a crawl is extremely long. Can you help me?
Technical SEO | | KnowHowww0 -
Shutting down a site, where do I 301 it?
I'm working with a few international sites that we are going to collapse into one main site. Our current plan is to 301 the 4 other sites into our main site home page. Is this ok? Is there a better way to do this? Thanks
Technical SEO | | MarloSchneider0 -
Pagination/Crawl Errors
Hi, Ive only just joined SEO moz and after they crawled my site they came up with 3600 crawl errors mostly being duplicate content and duplicate urls. After researching this it soon became clear it was due to on page pagination and after speaking with Abe from SEO mozhe advised me to take action by getting our developers to implement rel=”next” & rel=”prev” to review. soon after our developers implemented this code ( I have no understanding of this what so ever) 90% of my keywords I had been ranking for in the top 10 have dropped out the top 50! Can anyone explain this or help me with this? Thanks Andy
Technical SEO | | beck3980 -
Site Wide Links
I have a link on pr 3 home page website placed in the side bar. It is on a WordPress website that spans a couple hundred pages and the side bar is on every page. The majority of the pages are not ranked or have any pr. Can this affect me negatively?
Technical SEO | | raph39880 -
What is wrong with my site?
I have been working hard for over two months on my sites in seomoz and have seen some nice results in some (www.etraxc.com/ and www-my-etraxc.com for instance. Still I am really frustrated by www.classroomconnection.us/. I cant even get on the first page with the search term "classroom connection." i would love some help on this one. On a related note, does it help to have links to YouTube videos about the content? If so, how do I ensure that this piece is working well for me? Thanks a ton!
Technical SEO | | bobbabuoy0 -
.CA site same as .com site - are both necessary?
Dear Friend, We representa a major national brand in the auto care industry, and they have locations in both US and Canada. There is a primary content site at .com that we have duplicated at .ca. We are hosting the .ca site on a separate IP on a server in Canada - but by in large it is the same site. (there are some minor changes we made to change US English to Canadian English - though minor. When we search Google.ca we generally see strong search results for the .com site, but rarely, if ever any evidence of rankings for the .ca site. The .com site was launched several years ago about 18 months before the .ca site. Why doesn't Google.ca show the .ca site? Is this an issue of duplicate content, and Google.ca simply shows the .com version which it knew about first? Are we wasting our time, money and efforts having both? Thanks, Tim ps. this isn't about location. We use a separate site to locate local shops, and have coordinated that well with Google Places, and when looking for local auto care - we do well in both US and Canada. The sites described above are largetl content sites.
Technical SEO | | lunavista-comm0