Two Robots.txt files
-
Hi there
Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one.
Thanks in advance!
Cheers,
Satla -
Thank Riera
-
Hi Satia,
You mentioned that one robots.txt file placed in inside the server and search engine unable to see the file. If search engine won't see robots.txt file then what is the use of that robots.txt file?
AFAIK it must be placed under root directory an there is no way to keep two files with the same name. So you must have only one robots.txt and that should be placed under root directory.
Hope this helps.
Thanks
-
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
-
Hi,
I don't find any good reason why there are two files. There should be just one, where you specify everything you'd like to done.
If the tech team doesn't want to correct and leave just one file, may be because they are lazy or there might be some other issue that if they delete one file, the hole site blows up.Here, I leave you 2 Moz's articles about the robots.txt file.
What is Robots.txt? - Moz Learn
Learn About Robots.txt with Interactive Examples - Moz BlogTake into account that the name of the file must be in lower case. I've never seen any different and the servers are usually case sensitive with the filenames.
Hope its helpful.
GR
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need to change permalink structure - need advice on these two options
I have a client with a robust Wordpress blog that ranks well for a number of high-volume terms. We are doing a site rebuild, and they are adamant they don't want to lose any SERP ranking. However, their permalink structure contains the date the post was published, i.e. https://www.company.com/blog/2018/10/28/blog-post-title The date inclusion is not ideal for obvious reasons, but also because it's creating 404 links when pre-scheduling with Hubspot. Option 1: Change the permalink structure for all posts, past and future, and implement 301 redirects The only reason I'm not outright doing this is because there will be some slippage in the rankings, given that redirects only pass on a percentage of juice. The client doesn't want any slippage. Option 2: Create a new instance of Wordpress in a new subdirectory and continue the blog from there This will mandate a new structure for the blog posts; now posts will be children of /industry-blog/ instead of /blog/, but the old posts will remain as they are under /blog/ (and with dates intact). But I don't know if this will cause some hierarchical confusion that will negatively impact both the old blog and the new blog. Any advice given is appreciated. Please feel free to correct me if I've gotten anything wrong, I've only been practicing SEO for a few years now.
On-Page Optimization | | Marce5210 -
Robots.txt Question for E-Commerce Sites
Hi All, I have a couple of e-commerce clients and have a question about URLs. When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout I'm not sure that I want these indexed. Could I be causing any harm/danger if I add this to the robots.txt file? /*? Any suggestions welcome! Gavin
On-Page Optimization | | IcanAgency0 -
How google treats my two different domains with the same content ?
I have two internet stores for two different markets but in the same language (English), the same content and the same url (only domains different). They are in different servers one in USA another in UK. Example: sample.com (global) and sample.uk (for UK).
On-Page Optimization | | VaidasLinen
Currently sample.com (7 years old) is doing better but not very very well, sample.uk (2 years old) is rated poorly. My question is if it's possible that google will rank both stores well in the future ? Thanks Vaidas0 -
Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare). We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain. If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains? Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error! Cheers, Dave.
On-Page Optimization | | davelane.verve0 -
The need of two-keyword optimization in the same page
Hi there! Due to the business model of my company I have to optimize two keywords in one page. I just asked about this question before and someone told me thant as long as they refer to the same concept and have almost the same "meaning", it is possible. The problem is how to face it up. I mean, there's one H1 label, one title, etc....and what's the "policy" of key distribution in the content in order to priorice these keywords? What you guys recommend? many thanks
On-Page Optimization | | juanmiguelcr0 -
Is it best to optimize your site for just one or two keywords?
My company/website makes and sells a product that's not that competitive but still has about 20 key words/phrases that people search for. My site is not a huge site maybe 35 pages after you include the blog posts.We sell samples off the site but it's mostly used as a brochure but we also want it to be a successful tool at bringing in leads. Should I optimize for the most popular key word phrases focusing on only one or two per page and forget about the rest or should I try to optimize for as many keywords as possible on all pages or should I optimize for just the few (3-5) heavy hitting keywords but on all pages? Right now I've got it optimized for around 3 keyword phrases for the whole site and only 1 or 2 per page with the most popular phrases on the most important pages.
On-Page Optimization | | JAARON0 -
Two different keywords - one URL
We're new to SEO, but have two keywords that are really not quite the same, but Google has targeted the same URL for us ... which means that SEO Moz is recommending we optimize the same URL, for opposite keywords (using the on page SEO). For example, the keywords (these aren't our keywords) of say, "beer brewing" and "ways to make beer for small breweries" are both pointing at our home page. The on page SEO is showing that "beer brewing" is a rank of say, a google ranking of 9. However, "ways to ..." is a google ranking of 47. So ... what am I supposed to do now? Do I rewrite the page to have "ways to ..." more prominent? I cannot really have the title and h1's include both ... What do I do now? We have about 3 or 4 of these "pairs". -- Anthony
On-Page Optimization | | apresley0 -
Image files names - should each be different ?
Hi SEOs I have created a page on my website with a certain tutorial - let's say "How to change a tyre in your car". I have Grade A from On-page report using SEOmoz tool. I have put a relevant alt description for each photo I have put in that tutorial and I have used relevant keyword as a name for each image. So for example this would look like that: changing-tyre-1.jpg changing-tyre-2.jpg My question is should I name these file differently according to their content ? For example if on my photo there is some tool I have to use to change the tyre should the file have the name relevant to the subject of the article/tutorial on the page or to the content of the image ? Is that a stupid question ? Or am I getting too fussy ? Thanks
On-Page Optimization | | lolskizz0