New server update + wrong robots.txt = lost SERP rankings
-
Over the weekend, we updated our store to a new server. Before the switch, we had a robots.txt file on the new server that disallowed its contents from being indexed (we didn't want duplicate pages from both old and new servers).
When we finally made the switch, we somehow forgot to remove that robots.txt file, so the new pages weren't indexed. We quickly put our good robots.txt in place, and we submitted a request for a re-crawl of the site.
The problem is that many of our search rankings have changed. We were ranking #2 for some keywords, and now we're not showing up at all. Is there anything we can do? Google Webmaster Tools says that the next crawl could take up to weeks! Any suggestions will be much appreciated.
-
Dr. Pete,
I just ran across one of your webinars yesterday and you brought up some great ideas. Earned a few points in my book
Too often SEOs see changes in the rankings and react to counter-act the change. Most of the time these bounces are actually a GOOD sign. It means Google saw your changes and is adjusting to them. If your changes were positive you should see positive results. I have rarely found an issue where a user made a positive change and got a negative result from Google. Patience is a virtue.
-
Thanks everyone for the help! Fortunately we remedied the problem almost immediately, so it only took about a day to get our rankings back. I think the sitemap and fixed robots.txt were the most important factors.
-
I agree, let Google re-index first and then re evaluate the situation.
-
I hate to say it, but @inhouseninja is right - there's not a lot you can do, and over-reacting could be very dangerous. In other words - don't make a ton of changes just to offset this - Google will re-index.
A few minor cues that are safe:
(1) Re-submit your XML sitemap
(2) Build a few new links (authoritative ones, especially)
(3) Hit social media with your new URLs
All 3 are at least nudges to re-index. They aren't magic bullets, but you need to get Google's attention.
-
Remain calm. You should be just fine. It just takes time for Google to digest the new robots.txt. I would be concerned if things didn't change in 3-4 weeks. Adopt a rule to not freak out on Google until you've given the problem 14 days to resolve. Sometimes Google moves things around and this is natural.
If you want Google to crawl your site faster, build some links and do some social media. That will encourage Google to speed it up.
-
If this is all that happened the next crawl should fix it. Just sit tight and they should bounce up again in a week or so.
-
That does not sound fun at all.... So you just changed the server, complete copy?
My first question would be other than the server did anything else change? Copy or URL's
My second question would be is the other server still up and live to the internet?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Have a Robots.txt Issue
I have a robots.txt file error that is causing me loads of headaches and is making my website fall off the SE grid. on MOZ and other sites its saying that I blocked all websites from finding it. Could it be as simple as I created a new website and forgot to re-create a robots.txt file for the new site or it was trying to find the old one? I just created a new one. Google's website still shows in the search console that there are severe health issues found in the property and that it is the robots.txt is blocking important pages. Does this take time to refresh? Is there something I'm missing that someone here in the MOZ community could help me with?
Intermediate & Advanced SEO | | primemediaconsultants0 -
Why my site not ranking
Hello everyone, can anyone suggest me, where i am having problem in my site www.suntechengineers.com, i know content is less,
Intermediate & Advanced SEO | | poojathakar
but any other things that i am missing in my site? Is There any on page query please let me know, i need urgently getting up this,please help thanx in advance0 -
New Google Update on 21st April
Hi We all know that the mobile update is coming on the 21st April and if your site isnt mobile friendly in Googles eye you will be removed from the mobile index. Will this affect tablets. Most of our pages are mobile friendly but there are a few which arent. However these are tablet friendly. I havent heard Google mention about tablet rankings. Thanks Andy
Intermediate & Advanced SEO | | Andy-Halliday0 -
New site started ranking, lost ground after 301...?
Hi everyone. So we decided to re-brand a website starting with a fresh domain and fresh website (new everything). The first 4 weeks the site was performing really well and started bringing organic traffic for several targeted keywords (small amounts, but something). Around 5 weeks later, we decided to perform a 301 Redirect from an older site we had that really hadn't changed since 2007. As soon as we performed the 301, we started loosing ground on the keywords that were starting to perform. I may help to mention that we were targeting different keywords on the new site, versus the keywords/industry that were targeted on the older site... because we were focusing on another but similar industry. Now, 10 weeks later we are still not showing up for the keywords that we were starting to make headway on in the first four weeks of starting.... any ideas why? suggestions? The 301's were performed properly. We verified them, and we check Google WMT for any additional ones that may still be out there, but try to fix them as soon as possible. We have grown the site from just a few pages to over 60+ pages in the last 2 months with unique and fresh content targetting those keywords... Thank you in advance
Intermediate & Advanced SEO | | co.mc0 -
A First For Me: Client Wants New Website & Completely Updated Content
A client that I worked with on another project has approached me asking if I can handle the transition from his old website to his new website. He already had the new website designed and the URL structure is completely different than the old one. Normally this would not be a problem, just 301 redirect each page to the new page. However, this client has COMPLETELY redone his website from the ground up including navigation, pages and page content. The old website has been around for 8 years and is ranking for some good keywords. The reason he decided to build the new site is because the URL of his old domain is very long and for whatever reason he didn't like it (I'm assuming this would be misspellings of people trying to get back to his website and long email addresses, but he didn't clarify). I have never dealt with such a drastic change before and wanted the SEOMoz community input on the best way to pass authority/link juice from the old domain to the new one. Thanks in advance for your help.
Intermediate & Advanced SEO | | Bo-Jangles0 -
How many links would you need to rank up in page rank?
White hat **** Can 20 website with page rank of 3 make your site rank higher?
Intermediate & Advanced SEO | | spidersite0 -
Will blocking urls in robots.txt void out any backlink benefits? - I'll explain...
Ok... So I add tracking parameters to some of my social media campaigns but block those parameters via robots.txt. This helps avoid duplicate content issues (Yes, I do also have correct canonical tags added)... but my question is -- Does this cause me to miss out on any backlink magic coming my way from these articles, posts or links? Example url: www.mysite.com/subject/?tracking-info-goes-here-1234 Canonical tag is: www.mysite.com/subject/ I'm blocking anything with "?tracking-info-goes-here" via robots.txt The url with the tracking info of course IS NOT indexed in Google but IT IS indexed without the tracking parameters. What are your thoughts? Should I nix the robots.txt stuff since I already have the canonical tag in place? Do you think I'm getting the backlink "juice" from all the links with the tracking parameter? What would you do? Why? Are you sure? 🙂
Intermediate & Advanced SEO | | AubieJon0 -
New Domain name vs Low Ranked Existing Site
I am going to build a new site. I could hang it off an existing site with similar content or buy a new keyword rich domain and start over. The existing site does not have much trust or authority beyond the domain being registered for 5 plus years. I would prefer to start over and build linksfrom scratch but I realize we are starting at the bottom. The keywords we will be competing against are not super competetive so I think we can get ranking within 6 months or so. These post Panda days I am rethinking everything so any input is appreciated. I did a similar niche site a few years ago and found the site ranked well fairly quickly for its little nice. Today though it may be different. I have no experience in buying domains and would have no idea where to start there. New or existing? Thanks for any input.
Intermediate & Advanced SEO | | Reportcard0