Robots file include sitemap
-
Hello,
I see that google, facebook and moz... have robots.txt include sitemap at the footer.
Eg: http://www.google.com.vn/robots.txtSitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Should I include my sitemap file (sitemap.xml) at the footer of robots.txt and why should do this?Thanks,
-
Fore sure. The reason is that not all sitemap files are simply sitemap.xml it may be sitemap.gzip sitemap.zip or in my case, sitmap_index.gzip. Also, some people may not be able to include their sitemap at root.
Including sitemap exacts in robots.txt gives a clear directive to each of the search engines exactly where to find your sitemap. Google and Bing/Yahoo will have no issue finding it as you probably submitted it to them but crawlers like ask.com will usually look at your robots.txt and skip your site if no map is placed in it.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
HTML Visual Sitemap
Hello, Can anyone suggest the best tool for a visual HTML sitemap? I want to show what a website looks like architecturally before and be able to drag pages around visually to show an enhanced site architecture. I have looked at a few tools online but would like to try before I buy and also get recommendations. Any ideas?
On-Page Optimization | | AL123al1 -
Can you reference http and https in a sitemap.xml?
I have a site where some pages (in Spanish) are https. The english pages are http. Can you tell me if it's okay to have both in the sitemap.xml?
On-Page Optimization | | RoxBrock0 -
Ajax tabs and seo and sitemap
I am in the process of simplifying my website so that there are tabs in each section which load dynamically by ajax all of the subpages.Each subpage has funny url with a hashtag.
On-Page Optimization | | paddyaran
Does this mean each page does not have its title , or metatags?, Is this bad for seo?. Many of these pages have only a paragraph of text. I also am making a sitemap which links to the pages as a normal page. Would this mean I have a duplicate content issue with seo?.0 -
.htaccess file uploaded, website won't load
I uploaded the .htaccess file with the below, and now my website won't load at all? Then I deleted the htaccess file and it still won't load? But then it would load on my phone when I took it down, not on chrome, or explorer? Then I put it back up and looked again on my phone, wouldn't load on phone. Then deleted file and it still won't load on my phone? What is going on? RewriteEngine on
On-Page Optimization | | dwebb007
RewriteCond %{HTTP_HOST} !^http://freightetc.com$
RewriteRule ^(.)$ http://www.freightetc.com/$1 [R=301]
RewriteCond %{THE_REQUEST} ^./index.php
RewriteRule ^(.*)index.php$ http://www.freightetc.com/$1 [R=301]0 -
How important is it to include the target keyword phrase in the page URL?
If I want to target a keyword phrase to a particular phrase, but do not want to change the URL of that page, will that negatively impact my rankings? I am also wondering if I can get around it by creating a new, short URL that 301 redirects to the original URL. Would that be as effective as including the keyword in the original URL?
On-Page Optimization | | susannajbost0 -
XML sitemap latest best practices?
If anyone optimised their sitemap recently and would like to share some tips it would be much appreciated:-) Cheers
On-Page Optimization | | dublinbet0 -
On my site, www.myagingfolks.com, only a small number of my pages appear to be indexed by google or yahoo. Is that due to not having an XML sitemap, keywords, or some other problem?
On my site, www.myagingfolks.com, only a small number of my pages appear to be indexed by google or yahoo. I have thousands of pages! Is that due to not having an XML sitemap, keywords, or some other problem?
On-Page Optimization | | Jordanrg0 -
Include the company/domain name in page titles and urls?
I know this isn't something that I would use site-wide but I'm wondering if it helps or hurts me to use my company name (also my domain name) in pages below the homepage. As an example, let's say I'm Home Depot. In the category pages off the homepage should I use Page names and urls like Home and Garden Supplies or Home and Garden Supplies at Home Depot? Or does it hurt me to reuse my company/domain name on multiple pages?
On-Page Optimization | | kdieruf0