Robots.txt and 301
-
Hi Mozzers,
Can you answer something for me please.
I have a client and they have 301 re-directed the homepage '/' to '/home.aspx'.
Therefore all or most of the linkjuice is being passed which is great. They have also marked the '/' as nofollow / noindex in the Robots.txt file so its not being crawled.
My question is if the '/' is being denied access to the robots is it still passing on the authority for the links that go into this page? It is a 301 and not 302 so it would work under normal circumstances but as the page is not being crawled do I need to change the Robots.txt to crawl the '/'?
Thanks
Bush
-
Thanks Yannick - really helpful
cheers and have a good weekend
Bush
-
Yes. Remove the noindex nofollow for / from the robots.txt. It doesn't add anything. It can only confuse SE's and lose you rankings.
-
Hi Yannick.
Thanks for your reply. The / redirect to /home redirect has been done and the / is excluded.
I'm limited as to what changes I can make. I can amend the robots.txt to crawl / which is 301's to /home and that way the links will be attributed and the 301 remains.
Will that be beneficial do you think?
Bush
-
I would suggest not excluding the / in the robots.txt.
First you redirect the / to home.aspx but then you tell SE's to exclude the home page which you are redirecting anyway. Makes no sense to me, so I would remove it in the robots.txt. No use anyway, because the url is being redirected.
I wouldn't use the 301 redirect to home.aspx. Why not make some (technical?) modifications so that the home page is available on the root of the domain? It's much more beneficial to you site's SEO. Or the other way around, redirect home.aspx to /. Because way more ppl will be linking to the root and not the home.aspx page.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have two robots.txt pages for www and non-www version. Will that be a problem?
There are two robots.txt pages. One for www version and another for non-www version though I have moved to the non-www version.
Technical SEO | | ramb0 -
Clarification regarding robots.txt protocol
Hi,
Technical SEO | | nlogix
I have a website , and having 1000 above url and all the url already got indexed in Google . Now am going to stop all the available services in my website and removed all the landing pages from website. Now only home page available . So i need to remove all the indexed urls from Google . I have already used robots txt protocol for removing url. i guess it is not a good method for adding bulk amount of urls (nearly 1000) in robots.txt . So just wanted to know is there any other method for removing indexed urls.
Please advice.0 -
Is there any value in having a blank robots.txt file?
I've read an audit where the writer recommended creating and uploading a blank robots.txt file, there was no current file in place. Is there any merit in having a blank robots.txt file? What is the minimum you would include in a basic robots.txt file?
Technical SEO | | NicDale0 -
Another 301 redirect question - penalty?
Good Morning, We have 2 sites have images and minimal text on them. The images have links that point to a 3<sup>rd</sup> site that facilitates eCommerce. Question: If we 301 redirect these sites permanently to yet a 4<sup>th</sup> site… 1) Does it violate any G’s guidelines 2) Should we delete the links embedded in the images (as they point to the 3<sup>rd</sup> site) Thanks
Technical SEO | | Prime850 -
BEST Wordpress Robots.txt Sitemap Practice??
Alright, my question comes directly from this article by SEOmoz http://www.seomoz.org/learn-seo/robotstxt Yes, I have submitted the sitemap to google, bing's webmaster tools and and I want to add the location of our site's sitemaps and does it mean that I erase everything in the robots.txt right now and replace it with? <code>User-agent: * Disallow: Sitemap: http://www.example.com/none-standard-location/sitemap.xml</code> <code>???</code> because Wordpress comes with some default disallows like wp-admin, trackback, plugins. I have also read other questions. but was wondering if this is the correct way to add sitemap on Wordpress Robots.txt http://www.seomoz.org/q/robots-txt-question-2 http://www.seomoz.org/q/quick-robots-txt-check. http://www.seomoz.org/q/xml-sitemap-instruction-in-robots-txt-worth-doing I am using Multisite with Yoast plugin so I have more than one sitemap.xml to submit Do I erase everything in Robots.txt and replace it with how SEOmoz recommended? hmm that sounds not right. User-agent: *
Technical SEO | | joony2008
Disallow:
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-login.php
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments **ERASE EVERYTHING??? and changed it to** <code> <code>
<code>User-agent: *
Disallow: </code> Sitemap: http://www.example.com/sitemap_index.xml</code> <code>``` Sitemap: http://www.example.com/sub/sitemap_index.xml ```</code> <code>?????????</code> ```</code>0 -
301 redirect or 302
A client of mine has an international company named www.taxglobalizers.com if you come from Holland they 302 redirect you to the www.taxglobalizers.com/en version of the site and if you come from Holland they redirect with a 302 to www.taxglobalizers.com/nl version. I think it is best to change this in a 301 redirect. Am i correct in this? Kind Regards, Ruud
Technical SEO | | RuudHeijnen0 -
Quick robots.txt check
We're working on an SEO update for http://www.gear-zone.co.uk at the moment, and I was wondering if someone could take a quick look at the new robots file (http://gearzone.affinitynewmedia.com/robots.txt) to make sure we haven't missed anything? Thanks
Technical SEO | | neooptic0 -
How do I use the Robots.txt "disallow" command properly for folders I don't want indexed?
Today's sitemap webinar made me think about the disallow feature, seems opposite of sitemaps, but it also seems both are kind of ignored in varying ways by the engines. I don't need help semantically, I got that part. I just can't seem to find a contemporary answer about what should be blocked using the robots.txt file. For example, I have folders containing site comps for clients that I really don't want showing up in the SERPS. Is it better to not have these folders on the domain at all? There are also security issues I've heard of that make sense, simply look at a site's robots file to see what they are hiding. It makes it easier to hunt for files when they know the directory the files are contained in. Do I concern myself with this? Another example is a folder I have for my xml sitemap generator. I imagine google isn't going to try to index this or count it as content, so do I need to add folders like this to the disallow list?
Technical SEO | | SpringMountain0