My website pages are not crawled, what to do?
-
Hi all.
I have made some changes on the website so i like to crawled them by the search engines Google especially.
I have made these changes around 2 weeks ago. I have submitted my website on good bookmarking websites.
Also i used a tool available in Google webmasters "Fetch as Google", Resubmitted a sitemap.xml.
Still my pages are not crawled your opinion please.
Thanks
-
this is somenting I would love to know more!-)
I think news and updates, maybe a blog in your site can make google "feel" it has to come back more times to refresh your site again ( "hum dude, they are doing something there)... Links to your site and to new articles in social media could help a lot.
If you find anything let us know!-)
-
SEO Martin,
all these 93 results are index successfully also shows in our Google webmaster account.
But i am worried that why crawler is not visit our website pages on regular basis as we submit sitemap.xml in webmaster account after the changes have made.
what is the reason behind this.
also can you please explain me that what the other methods to force the search engine crawler to crawl our page.
Regards
Lucid
-
Hi Sha,
I just add this meta tag (as below) to help to crawl the page but not sure how it is.
I am just worried that if we make changes in our website pages and submit the site map in webmaster (that's mean that we are put request to search engine crawler to crawl the page ) but why they do not.
what is the actual reason behind this.
also can you please explain me that what the other methods to force the search engine crawler to crawl our page.
Regards
Lucid
-
HI Martin,
Exactly, all url are indexed in search engines but they are not crawled after making a changes that is the main concern for me now.
For example:
Take a look on Magento Development page that showing last crawled 17 Mar 2013. I have made some changes and that should be crawled on a regular basis same as previous crawling.
Url :http://www.lucidsoftech.com/magento-development.html
your thoughts?
Thanks
-
Lucid,
I´ve just made a search on google and I found 93 results for your site, http://www.lucidsoftech.com/ , is that url right?
How many results do you actually own?
-
HI Sha,
yes site is indexed and crawled on regular basis before making these changes. don't know why not this time and not crawled around 2 weeks. yes i checked for thart all have including tag.
Normally when u submitted your pages on few good social bookmarking, workout in webmasters, after generate few good back links all these thing are very helpful for page crawling.
Still waiting for the page crawling.Thanks Sha
-
HI Martin,
Before making these changes they crawled regular basis and I just changed a meta title keywords and description but don't know the reason why after making these changes on few pages my web pages are not crawled in past 2 weeks and and still they are not crawled.
Martin i have my robots.txt in public.html and i allow the page that should be crawled by the bots.
waiting fro the page crawling.Thanks
-
Hi lucidsoftech,
The site is definitely indexed, but without knowing the specific URLs you are waiting on, it is a little difficult.
I ran a quick crawl with Screaming Frog and there are actually quite a few pages on the site which have the noindex meta tag in place - have you checked that the pages you are wanting indexed are not excluded with a tag?
Just a thought,
Sha
-
make sure you have a robots.txt on public folder. You must allow bots and disallow some folders.
Example of robots.txt of wordpress site
User-agent: *
Allow: /Disallow: /*?
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Disallow: /tmpAllow: /wp-content/uploads/
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO Content Audits Questions (Removing pages from website, extracting data, organizing data).
Hi everyone! I have a few questions - we are running an SEO content audit on our entire website and I am wondering the best FREE way to extract a list of all indexed pages. Would I need to use a mix of Google Analytics, Webmaster Tools, AND our XML sitemap or could I just use Webmaster Tools to pull the full list? Just want to make sure I am not missing anything. As well, once the data is pulled and organized (helpful to know the best way to pull detailed info about the pages as well!) I am wondering if it would be a best practice to sort by high trafficked pages in order to rank them for prioritization (ie: pages with most visits will be edited and optimized first). Lastly, I am wondering what constitutes a 'removable' page. For example, when it is appropriate to fully remove a page from our website? I understand that it is best, if you need to remove a page, to redirect the person to another similar page OR the homepage. Is this the best practice? Thank you for the help! If you say it is best to organize by trafficked pages first in order to optimize them - I am wondering if it would be an easier process to use MOZ tools like Keyword Explorer, Page Optimization, and Page Authority to rank pages and find ways to optimize them for best top relevant keywords. Let me know if this option makes MORE sense than going through the entire data extraction process.
Technical SEO | | PowerhouseMarketing0 -
Using Product Page Content from an Offline Website
Hi all, We have two websites. One of the website's no longer sells product range A. However, on the second website, we would like to sell range A. We paid a copywriter to write some really good content for these ranges and we were wondering if we would get stung for duplicate content if we took these descriptions from website 1 and placed them on website 2. The products / descriptions are live anymore and haven't been for about 6 weeks. We're ranking for some great keywords at the moment and we don't want to spoil that. Thanks in advance! D
Technical SEO | | 10dales0 -
Google not crawling the website from 22nd October
Hi, This is Suresh. I made changes to my website and I see that google is unable to crawl my website from 22nd October. Even it is not showing any content when I use Cache:www.vonexpy.com. Can any body help me in knowing why Google is unable to crawl my website. Is there any technical issue with the website? Website is www.vonexpy.com Thanks in advance.
Technical SEO | | sureshchowdary1 -
Changes to website haven't been crawled in over a month
We redesigned our website at http://www.aptinting.com a few months ago. We were fully expecting the crawl frequency to be very low because we had redesigned the website from a format that had been very static, and that probably has something to do with the problem we're currently having. We made some important changes to our homepage about a month ago, and the cached version of that page is still from April 2nd. Yet, whenever we create new pages, they get indexed within days. We've made a point to create lots of new blog articles and case studies to send a message to Google that the website should be crawled at a greater rate. We've also created new links to the homepage through press releases, guest blog articles, and by posting to social media, hoping that all of these things would send a message to Google saying that the homepage should be "reevaluated". However, we seem to be stuck with the April 2nd version of the homepage, which is severely lacking. Any suggestions would be greatly appreciated. Thanks!
Technical SEO | | Lemmons0 -
Duplicate Page Title Crawl Error Issue
In the last crawl for on of our client websites the duplicate page title and page content numbers were very high. They are reading every page twice. http://www.barefootparadisevacations.com and http://barefootparadisevacations.com are being read as two different pages with the same page title. After the last crawl I used our built in redirect tool to redirect the urls, but the most recent crawl showed the same issue. Is this issue really hurting our rankings and if so, any suggestions on a fix for the problem? Thank you!
Technical SEO | | LoveMyPugs0 -
Local Search | Website Issue with Duplicate Content (97 pages)
Hi SEOmoz community. I have a unique situation where I’m evaluating a website that is trying to optimize better for local search and targeting 97 surrounding towns in his geographical location. What is unique about this situation is that he is ranking on the 1st and 2nd pages of the SERPs for his targeted keywords, has duplicate content on 97 pages to his site, and the search engines are still ranking the website. I ran the website’s url through SEOmoz’s Crawl Test Tool and it verified that it has duplicate content on 97 pages and has too many links (97) per page. Summary: Website has 97 duplicate pages representing each town, with each individual page listing and repeating all of the 97 surrounding towns, and each town is a link to a duplicate page. Question: I know eventually the site will not get indexed by the Search Engines and not sure the best way to resolve this problem – any advice?
Technical SEO | | ToddSEOBoston0 -
Why is my website banned?
IMy website is Costume Machine at www.costumemachine.com . My site has been banned for 1 year now. I have requested that google reconsider my site 3 times without luck. The site is dynamic and basically pulls in feeds from affiliate sites. We have added over 1,500 pages of original content. The site has been running great since 2008 without any penalties. I don't think I got hit with any linking penalty. I cleaned up all questionable links last November when the penalty hit. Am I being hit with a "thin" site penalty? If that is the issue what is the best way to fix the problem?
Technical SEO | | tadden0 -
Google crawl index issue with our website...
Hey there. We've run into a mystifying issue with Google's crawl index of one of our sites. When we do a "site:www.burlingtonmortgage.biz" search in Google, we're seeing lots of 404 Errors on pages that don't exist on our site or seemingly on the remote server. In the search results, Google is showing nonsensical folders off the root domain and then the actual page is within that non-existent folder. An example: Google shows this in its index of the site (as a 404 Error page): www.burlingtonmortgage.biz/MQnjO/idaho-mortgage-rates.asp The actual page on the site is: www.burlingtonmortgage.biz/idaho-mortgage-rates.asp Google is showing the folder MQnjO that doesn't exist anywhere on the remote. Other pages they are showing have different folder names that are just as wacky. We called our hosting company who said the problem isn't coming from them... Has anyone had something like this happen to them? Thanks so much for your insight!
Technical SEO | | ILM_Marketing
Megan0