Two Robots.txt files
-
Hi there
Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one.
Thanks in advance!
Cheers,
Satla -
Thank Riera
-
Hi Satia,
You mentioned that one robots.txt file placed in inside the server and search engine unable to see the file. If search engine won't see robots.txt file then what is the use of that robots.txt file?
AFAIK it must be placed under root directory an there is no way to keep two files with the same name. So you must have only one robots.txt and that should be placed under root directory.
Hope this helps.
Thanks
-
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
-
Hi,
I don't find any good reason why there are two files. There should be just one, where you specify everything you'd like to done.
If the tech team doesn't want to correct and leave just one file, may be because they are lazy or there might be some other issue that if they delete one file, the hole site blows up.Here, I leave you 2 Moz's articles about the robots.txt file.
What is Robots.txt? - Moz Learn
Learn About Robots.txt with Interactive Examples - Moz BlogTake into account that the name of the file must be in lower case. I've never seen any different and the servers are usually case sensitive with the filenames.
Hope its helpful.
GR
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help recover lost traffic (70%) from robots.txt error.
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change. Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated. On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL? Thanks,
On-Page Optimization | | akin671 -
Image File Names for eCommerce?
Hi everyone! I'm wondering about naming my product photo file names for an E-Commerce site. Let's say I say have product named Abe Lincoln in the **Print **category for sale with 4 images, relatively similar but from different views for example.Could I name them as follows? 1) abe-lincoln-print.jpg 2) abe-lincoln-print-side-view.jpg 3) abe-lincoln-print-close-up.jpg 4) abe-lincoln-print-font-view.jpg Or is that too many keywords for the page? Should I be worried about keyword stuffing? Plus once I add in title and alt tags and descriptions this could also increase the keyword count for "abe lincoln print"?
On-Page Optimization | | TheFlyingSweetPotato0 -
"translation" of code in htaccess file
Hi everyone! I am a newbie to the whole SEO and html thing and I am trying to get a better understanding of the "behind the scenes" part of my website. I hope I can find someone here who can translate a piece of code for me that I have in my htaccess file: Options -Multiviews
On-Page Optimization | | momof4
Options +FollowSymLinks
rewritecond $1 !^(index.php|public|tmp|robots.txt|template.html|favicon.ico|images|css|uploads)
rewritecond %{REQUEST_FILENAME} !-f
rewritecond %{REQUEST_FILENAME} !-d
rewriterule ^(.*)$ index.php?link=$1 [NC,L,QSA] I know that something is getting redirected to the index file, but what (or when) exactly? Does the word "robots"mean that search engine crawlers are getting redirected here? And is this good or bad (in terms of SEO)? Or is this redirecting people who try to get to my robots/ template or image files?? Thanks in advance for any answers!0 -
Disavow Tool Submitting 2nd File
Hi About 2 Months ago I submitted a Disavow File using the Disavow Tool I have collected more links and I am ready to upload a 2nd File, However should I download the previous file in Webmaster Tools (Disavow Tool) and add these new links to that File or if I just upload and override the existing file with this file containing new links only will that be ok. What I dont want to do is do something that removes all the previous findings from the list so that google cannot see them anymore. I guess what I am trying to say is does Google just refer to the live file I am updating / overriding or once I have submitted a file weather i remove it or not will google still have a record of it and be referring to it ? Thanks Adam
On-Page Optimization | | AMG1000 -
Is it a problem to have the same phone number on two websites in same niche?
is it a problem to have the same phone number on two websites in same niche? For instance if I had two completely different websites for one painting contractor client. All the content is completely different other than the logo and phone. It is in the same niche for "spokane painters and spokane painting contractor" .. will google see the same phone and penalize?
On-Page Optimization | | Superflys0 -
Do I need a robots meta tag on the homepage of my site?
Is it recommended to include on the homepage of your site site? I would like Google to index and follow my site. I am using WordPress and noticed my homepage is not including this meta tag, therefore wondering if I should include it?
On-Page Optimization | | asc760 -
Best practice for Meta-Robots tag in categories and author pages?
For some of our site we use Wordpress, which we really like working with. The question I have is for the categories and authors pages (and similiar pages), i.e. the one looking: http://www.domain.com/authors/. Should you or should you not use follow, noindex for meta-robots? We have a lot of categories/tags/authors which generates a lot of pages. I'm a bit worried that google won't like this and leaning towards adding the follow, noindex. But the more I read about it, the more I see people disagree. What does the community of Seomoz think?
On-Page Optimization | | Lobtec0 -
Image files names - should each be different ?
Hi SEOs I have created a page on my website with a certain tutorial - let's say "How to change a tyre in your car". I have Grade A from On-page report using SEOmoz tool. I have put a relevant alt description for each photo I have put in that tutorial and I have used relevant keyword as a name for each image. So for example this would look like that: changing-tyre-1.jpg changing-tyre-2.jpg My question is should I name these file differently according to their content ? For example if on my photo there is some tool I have to use to change the tyre should the file have the name relevant to the subject of the article/tutorial on the page or to the content of the image ? Is that a stupid question ? Or am I getting too fussy ? Thanks
On-Page Optimization | | lolskizz0