Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Using one robots.txt for two websites
I have two websites that are hosted in the same CMS. Rather than having two separate robots.txt files (one for each domain), my web agency has created one which lists the sitemaps for both websites, like this: User-agent: * Disallow: Sitemap: https://www.siteA.org/sitemap Sitemap: https://www.siteB.com/sitemap Is this ok? I thought you needed one robots.txt per website which provides the URL for the sitemap. Will having both sitemap URLs listed in one robots.txt confuse the search engines?
Technical SEO | | ciehmoz0 -
Block Domain in robots.txt
Hi. We had some URLs that were indexed in Google from a www1-subdomain. We have now disabled the URLs (returning a 404 - for other reasons we cannot do a redirect from www1 to www) and blocked via robots.txt. But the amount of indexed pages keeps increasing (for 2 weeks now). Unfortunately, I cannot install Webmaster Tools for this subdomain to tell Google to back off... Any ideas why this could be and whether it's normal? I can send you more domain infos by personal message if you want to have a look at it.
Technical SEO | | zeepartner0 -
Robots.txt file
How do i get Google to stop indexing my old pages and start indexing my new pages even months down the line? Do i need to install a Robots.txt file on each page?
Technical SEO | | gimes0 -
My Website Ranking is terribly drop
So sorry I really need your all help 😞 I'm really frustrated and stressed. For my website http://www.cma-academy.edu.sg , it has been ranking on Google 1st page (with the keywords "design course/design courses") for the past over 6 months and the traffic was really good... But it has been almost 2 months since the big drop... my ranking all is gone in just a week and now it's even not on the top 50.. While my competitors websites are stand stil. 😞 I seriously really want to cry because of this.. And I'm sure there is nothing much wrong with my SEO to be abandoned by Google like this and Google still index my webpage, but the crawling rate is so much slow than before.. I really need some help from the expert, especially from seomoz.. Please help me in this, I'm willing to answer any questions as well...
Technical SEO | | AngieDang0 -
Should I use these Meta Tags or Remove it?
Hi, I have a lot of older pages that I am cleaning up older pages, and I see that I have <title>Actual Title</title> (I understand the importance of this tag.) (I have some text in this meta tag on a lot of pages, sometimes matching my title tag exactly but in some cases I treated it like a mini description. Should I remove the on my pages, or keep it and make sure it is the exact as the main Title Tag. -------- Question about meta tag #2. I have heard rumors that the keywords tag should be removed. example: Thanks in advance! Force7
Technical SEO | | Force70 -
Best Practice to Remove a Blog
Note: Re-posting since I accidentally marked as answered Hi, I have a blog that has thousands of URL, the blog is a part of my site. I would like to obsolete the blog, I think the best choices are 1. 404 Them: Problem is a large number of 404's. I know this is Ok, but makes me hesitant. 2. meta tag no follow no index. This would be great, but the question is they are already indexed. Thoughts? Thanks PS A 301 redirect to the main page would be flagged as a soft 404
Technical SEO | | Bucky0 -
Spam Backlinks to My Website
today i have created inbound link report using Link Research & Analysis tool and i found that there are number of spam inbound link to my website from lots of blogs and other sites Which anchor text are not relevant to my site. It contain some abusive words in anchor text like "viagra expiration date" and other. I want remove these irrelevant backlinks. As there are very high number of links approx 9000, its almost impossible to remove the links manually. Is there any way to remove and restrict those backlink? Whats steps required to protect any negative affect to my website? Please advice asap.
Technical SEO | | saupari0 -
Is SEO affected when hosting service is changed?
I put my plan to change web hosts on abeyance when an SEO friend of mine told me that it would negatively impact the sites SEO. I am not changing the url or anathing, just the host.
Technical SEO | | ambisuite0