Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Measuring the size of a competitors website?
I think our website is too big, far too many indexed pages. I'd like to do some research on how big our competitors' websites are (how many indexed pages). Is there a way to do this? Cheers, Rhys
Technical SEO | | SwanseaMedicine0 -
Old pages - should I remove them from serps?
Hi guys, I need an advice from you, a recommendation. I have some old LPs from old campaigns, around 70 pages indexed on Google, campaigns that are not available anymore. I have removed them from my DB, but they still remained on server so Google still sees them as URLs on my site, witch I totally agree. What should I do with this pages? Should I remove them completely? (url removal tool) or use rel=canonical? How will this affect my domain authority and rankings? This pages doesn't bring traffic any more, maybe a view now and then, but overall this pages don't bring traffic.
Technical SEO | | catalinmoraru0 -
Hosting Multiple Websites
I want host multiple websites in geo-specific locations, and also have them be unique from each other. Does anyone have suggestions of a software or another method to use for this? Any experience with http://multiplecloud.com?
Technical SEO | | theLotter0 -
Is there any value in having a blank robots.txt file?
I've read an audit where the writer recommended creating and uploading a blank robots.txt file, there was no current file in place. Is there any merit in having a blank robots.txt file? What is the minimum you would include in a basic robots.txt file?
Technical SEO | | NicDale0 -
Does bait and switch redirecting affect SEO?
Yeah, the question might be an odd one, but I'm new to SEO. Currently we've been considering creating links to certain pages on our site that instantly redirect to a landing page like this (referring to it as bait and switch redirecting): http://www.ecornell.com/certificate-programs/human-resources-management-training/advanced-certificate-in-strategic-hr-management/crt/ILRASHRC1/?lp=http://info.ecornell.com/executive-certificate-in-hr-leadership%3Fls=xxx%26of=xxx This essentially causes the user to "hop" from the content page directly to the landing page. Now, the content page is indexed otherwise, so I'm wondering if time on page (which is basically 0) or anything else might affect page rank. Also, if this is something we shouldn't do, that'd be great to know too. It feels sort of dirty, but I can't find any resources telling me it's bad, or how it would affect pagerank. I've read about pagejacking, but this doesn't seem to fall into the same category. Thanks! Alan
Technical SEO | | amccarty0 -
Websites not being included on google from mobiles?
Hi, Just had a call from a guy saying that google have made a statement saying that it will be stopping people finding websites from mobule devices if they dont have a mobile domain name. Doesn anyone know anything about any Google statements or is this just rubbish?
Technical SEO | | Ant710 -
Can a site be removed from alexa?
let's say you have complete control over the webserver, and the hosting server. is there a way to set it up so that alexa statistics CANNOT be gained?
Technical SEO | | highersourcesites0