Robots.txt Showing in SERP Results
-
Currently doing a technical audit for a website and when I search "Site:website.com -www" the only result is website.com/robots.txt
I was wondering if anyone else has come across this before -- or what this may mean from a technical audit standpoint.
Thank you!
-
nonsense. Search for https://www.google.com.au/search?q=inurl%3Arobots.txt&pws=0
Some of the first results with visible robots.txt I see are:
I refuse to believe that "something is seriously wrong" with any of these sites.
-
It's quite common for Google to index robots.txt files. (and also, rather odd) But check out all of these robots.txt files:
https://www.google.com/search?q=inurl%3Arobots.txt&pws=0&gl=us
So it's nothing to be alarmed by. With your particular query. "Site:website.com -www" it only shows pages indexed without the "www" so this just says that all the indexed pages most likely begin with www. The exception, of course, is the robots.txt file.
The bigger question for me is, why does Google cache robots.txt files? Oh well.
-
The robots.txt file should not show up. Sounds like there is something seriously wrong.
-
Did you also search for site:www.webiste.com? Are they blocking the site?What's in the actual robots file?
Also, does this happen when you search for the site in Bing?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Wrong domain name showing in the SERP
I work with Axos Bank, and the domain name in the SERP started showing axos-services.com a couple of weeks ago. *As a test, search for [axos bank] in Google. You may need to click on "Search instead for axos bank" under "Showing results for axis bank" which is a whole separate issue we want to resolve. They do own axos-services.com, and it redirects to axosbank.com (along with a number of other domains). The SERP has always shown axosbank.com until recently, but the redirect has been in place for some time. Any idea why and how to prompt Google to show the correct domain? Thanks, all!
Technical SEO | | Gildej0 -
Are all tabs the result of JS?
Hey all, Bit of a novice here, so bare with me. We have a page that has a lot of content in a tabular format that struggles to rank. I created a similar page, without the tabular format, which vastly outranks it, despite having a miniscule backlink profile in comparison. Now, I've always been under the impression that anything interactive on a website, like tabs, are the result of JS. However, I can't see any JS in the code (but as mentioned, I'm far from an expert). Code below. Description insert copy here, click on [insert anchor text](/media/MODULES AVAILABLE 2017-18.xlsx)
Technical SEO | | SwanseaMedicine
more copy here. Anyone able to shed any light? Cheers, Rhys0 -
No descripton on Google/Yahoo/Bing, updated robots.txt - what is the turnaround time or next step for visible results?
Hello, New to the MOZ community and thrilled to be learning alongside all of you! One of our clients' sites is currently showing a 'blocked' meta description due to an old robots.txt file (eg: A description for this result is not available because of this site's robots.txt) We have updated the site's robots.txt to allow all bots. The meta tag has also been updated in WordPress (via the SEO Yoast plugin) See image here of Google listing and site URL: http://imgur.com/46wajJw I have also ensured that the most recent robots.txt has been submitted via Google Webmaster Tools. When can we expect these results to update? Is there a step I may have overlooked? Thank you,
Technical SEO | | adamhdrb
Adam 46wajJw0 -
Do i have my robots.txt file set up properly
Hi, just doing some seo on my site and i am not sure if i have my robots file set correctly. i use joomla and my website is www.in2town.co.uk. here is my robots file, does this look correct to you User-agent: *
Technical SEO | | ClaireH-184886
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/ many thanks1 -
Titles in google serps incorrect
If you do the following query in google: site:orlandovisiting.com legend sparrow You'll see in the results the title appears as: legend-captain-jack-sparrow-1149.html Where the title tag in the source is: <title>The Legend of Captain Jack Sparrow Opens at Disney’s Hollywood Studiostitle> Eventually it normally rights itself, but does anyone else get this with their sites? Thanks.
Technical SEO | | walshy990 -
Server crashed - What should I do regarding Google SERP´s?
We have several travel websites in Uruguay since 2003. These sites have a very high PR and Trust. The server where all our sites are hosted has crashed and we have been for 2 days now trying to fix all this mess. We hope this problem will be fixed today. Please I really need to know what should I do regarding Google. I mean one of our sites has been ranking in top 1 positions for more than 150 keywords. Will we loose all that? What can we do about it? It´s the first time this has happened to our sites.
Technical SEO | | ceci27100 -
No results with Link Analysis
So I have been working with a domain since November last year that still shows no improvement in regards to the link analysis. I am baffled because we have gotten them onto the first page on Google for a few of the keywords we are optimizing. Any help with this is greatly appreciated and I am a noob so definitely open to learning. Thanks in advance to all of you. Domain in question - www.modernportablerefrigeration.com Domain is currently on a shared server if that makes any difference. Cordially, Todd Richard [email protected]
Technical SEO | | RichFinnSEO0 -
Robots.txt question
I want to block spiders from specific specific part of website (say abc folder). In robots.txt, i have to write - User-agent: * Disallow: /abc/ Shall i have to insert the last slash. or will this do User-agent: * Disallow: /abc
Technical SEO | | seoug_20050