Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Could the EMD update affect my domain?
My domain is: http://www.freescrabbledictionary.com/ "Scrabble Dictionary" is a huge keyword in my niche where I used to rank top 4. Do you see this domain as possibly being hit by the EMD? My Google Analytics does not show that I was initially hit back in Sept 2012 when it first same out.
Technical SEO | | cbielich0 -
Amazing results even after 1 week!
Have to say, I'm pretty impressed with Moz, this is now my first full week of membership and wow have I seen some great increases in my site stats! Hopefully this isn't just a blip and that it wil continue for weeks and months to come. Authority has jumped from 27 to 34 Google page one results jumped from 7 to 13 Trafffic increased by 12% Solved duplicate content issues Started a proactive social media campaign I could go on and on, but can't say enough positive things about the services that are provided here, an investment well worth paying and already paying for itself. The goal for the next few weeks is to improve domain authority from 34 to 40+, I've been using long tail phrases for my articles, which has been tremendously beneficial. One query is that even though the domain authority has moved from 27 - 34, I don't appear to have gained any extra backlinks - perhaps I'm misunderstand this metric? The other query is that there are 100's of backlinks pointing to my domain (I provide an open source cms so I know the links are there), but none of these lnks appear to be counting towards my authority. Is there a way I can submit these pages to the index on their behalf? Cheers, Lee
Technical SEO | | LeeC0 -
Should I block Map pages with robots.txt?
Hello, I have a website that was started in 1999. On the website I have map pages for each of the offices listed on my site, for which there are about 120. Each of the 120 maps is in a whole separate html page. There is no content in the page other than the map. I know all of the offices love having the map pages so I don't want to remove the pages. So, my question is would these pages with no real content be hurting the rankings of the other pages on our site? Therefore, should I block the pages with my robots.txt? Would I also have to remove these pages (in webmaster tools?) from Google for blocking by robots.txt to really work? I appreciate your feedback, thanks!
Technical SEO | | imaginex0 -
Meta-robots Nofollow
I don't understand Meta-robots Nofollow. Wordpress has my homepage set to this according to SEOMoz tool. Is this really bad?
Technical SEO | | hopkinspat1 -
Can I remove 301 redirects after some time?
Hello, We have an very large number of 301 redirects on our site and would like to find a way to remove some of them. Is there a time frame after which Google does not need a 301 any more? For example if A is 301 redirected to B, does Google know after a while not to serve A any more, and replaces any requests for A with B? How about any links that go to A? Or: Is the only option to have all links that pointed to A point to B and then the 301 can be removed after some time? Thank you for you you help!
Technical SEO | | Veva0 -
Website redesign launch
Hello everyone, I am in the process of having my consulting website redesigned and have a question about how this may impact SEO. I will be using the same URL as I did before, just simply replacing an old website with a new website. Obviously the URL structure will change slightly since I am changing navigation names. Page titles will also change. Do I need to do anything special to ensure that all of the pages from the old website are redirected to the new website? For example, should I do a page level redirect for each page that remains the same? So that the old "services" page is pointed to the new "services" page? Or can I simply do a redirect at the index page level? Thank you in advance for any advice! Best, Linda
Technical SEO | | LindaSchumacher0 -
Websites not being included on google from mobiles?
Hi, Just had a call from a guy saying that google have made a statement saying that it will be stopping people finding websites from mobule devices if they dont have a mobile domain name. Doesn anyone know anything about any Google statements or is this just rubbish?
Technical SEO | | Ant710 -
Leaving Comments on blogs when html is removed
I found the following blog. It is pagerank 5 do follow http://www.unssc.org/web1/programmes/rcs/cca_undaf_training_material/teamrcs/forumdetail.asp?ID=32 If you attempt to leave a comment with html, the html is removed. There is a button which allows you to leave a comment but if you do it gets redirected to the domain of the blog not your site. However there are still people leaving links with the url of the intended site. As late as today. look at this comment
Technical SEO | | mickey11
Comment posted by : Alex on 09/09/2011 I love to se percorsi on this site very often How is this done, if anyone knows I got the code done to this your keywords The important part being mce_real_href0