Need Third Party Input. Our Web host blocked all bots including Google and myself because they believe SEO is slowing down their server.
-
I would like some third party input... partly for my sanity and also for my client.
I have a client who runs a large online bookstore. The bookstore runs in Magento and the developers are also apparently the web host. (They actually run the servers.. I do not know if they are sitting under someones desk or are actually in a data center)
Their server has been slowed down by local and foreign bots. They are under the impression my SEO services are sending spammer bots to crawl and slow down their site.
To fix the problem they disallowed all bots. Everything, Google, Yahoo, Bing. They also banned my access from the site. My clients organic traffic instantly took a HUGE hit. (almost 50% of their traffic is organic and over 50% is Organic + Adwords most everything from Google)
Their keyword rankings are taking a quick dive as well.
Could someone please verify the following as true to help me illustrate to my client that this is completely unacceptable behavior on part of the host.
I believe:
1.) You should never disavow ALL robots from your site as a solution for spam. As a matter of fact most of the bad bots ignore robots.txt anyways. It is a way to limit where Google searches (which is obviously a technique to be used)
2.) On site SEO work as well as link building, etc. is not responsible for foreign bots and scrappers putting a heavy load on the server.
3.) Their behavior will ultimately lead to a massive loss of rankings (already happening) and a huge loss of traffic (already happening) and ultimately since almost half the traffic is organic the client could expect to lose a large sum of revenue from purchases made by organic traffic since it will disappear.
Please give your input and thoughts. I really appreciate it!
-
Thanks so much for your response. Glad to hear that there was a fairly good ending to this, and thanks for following up!
-
Keri -
I was able to produce multiple reports that accomplished the following:
1.) Illustrate the quick/graphic drop in our Google rankings
2.) Illustrate that the majority of traffic comes from Organic Search
3.) Tie together the trend that was already happening of a dramatic drop in organic traffic as keywords were slipping.
4.) Bring a reality to the fact that this behavior will quickly result in such a steep financial hit it was an 'emergency'.
In this particular situation there is My Client, Their Developers, Me the SEO guy.
I found that by helping my client to understand the situation, financial impacts, and why we had to act helped to spur things along.
We had a big meeting with everybody involved. This was a great opportunity to understand what the developers (Who also serve as the host) were going through. Facing attacks from bots, trying to keep the server alive, etc. etc. Ultimately it was revealed that they had a bug in their code for Magento that was causing a lot of extra DB hits that was a main root cause of their issues.
We were able to work out the following ground rules:
1.) NEVER block all bots under any circumstance
2.) The majority of our Organic traffic is Google then Bing/Yahoo were a tiny fraction and everything else didn't matter. I crafted a good robots.txt that let in all of the major bots I wanted and excluded the rest. Ideally I'd like to include most all of them. (Since we're only blocking good bots because the bad ones will just ignore) However, I wanted to compromise and also help them with server traffic. (PLUS for us Google is it.) I did make sure my robots.txt allowed in all Google services, etc.
3.) We set up a system to make sure everybody was in the loop when a dramatic decision regarding the website was made. (that's way better than me finding out a few days later that Google was blocked and damage has already been done)
4.) We really brought into light that SEO has/had nothing to do with the situation.
In the end the developers are great people but like everything else... they almost need to see you in person and hear why they can't do stuff like that. In their world it makes total sense because the server is overloaded. However, there won't be an overloaded server if you block out Google and all the traffic it sends.
We were able to recover most of our rankings and our traffic returned back to normal. We aren't quite back to where we were but getting there. The keywords snapped back fairly quickly but the organic traffic didn't so it might be something else. I actually will throw in a screenshot of the incident down below.
Thanks for checking up on it!
-
Hi Joshua, I'm looking through some older threads, and wondering if you're able to give us any type of update as to what happened in this case (and if you have any hair left!). I've had some battles with developers before too, and have sympathy for your position.
-
Thanks for your answers and help everyone! I really appreciate all of the details. I see the power of this community and hope to be able to contribute instead of only take in the future. Thanks again!!
-
Drop them as a client. They're paying you for SEO help but they obviously don't trust/like it. Not worth your time.
-
Quite clearly, this is bonkers.
If you block access from search engine spiders how can they possibly index the content in good faith? You are hoping that they will not crawl the content to check what is there (or burn server resources) but they will still happily refer users of their search engine to these pages in good faith.
Additionally, it is highly unlikely that bots from the major engines are causing a measurable impact - Google for instance states they will only crawl one page every few seconds (1).
That said, there are a lot of parasites out there and crawlers that will eat up server time so there may also be some truth in what the host is saying. That said, there is still no excuse for this hatchet job of sorting things out.
The other angle here is that magento and ecommerce sites can often be a crawlers worst nightmare. As an example product comparison systems can often create thousands (I have seen millions) of crawlable URLs - now a sensible spider has a crawl budget and will give up but that's not saying all will. A simple crawl in screaming frog should give you an idea here (not that you will be able to do that) and in many cases where these problems exist this is enough to bring a server to it's knees.
In my mind you have a few things to do here
1. Convince the host that blocking all spiders is incorrect
Hopefully this thread and the references here should be more than enough to do this. Beyond that simply show them a fetch as Googlebot & the Crawl section in webmaster tools and you should be able to make your point quickly and easily.
2. Help the developer implement a more sensible list of what to block.
This article is a good start here:
http://searchenginewatch.com/article/2067357/Bye-bye-Crawler-Blocking-the-ParasitesRemember you can allow one (or more) robots and then disallow everything else:
User-agent: Google Disallow: User-agent: * Disallow: /
Other options also exist such as limiting the speed at which a crawler can crawl - well, requesting that they limit the speed at which they crawl.
Also remember that any truly parasitic bot or crawler will likely ignore robots.txt anyway so you may need to implement some more advanced blocking at a firewall or server level.
3. Help the developer identify the cause of the resources problem
As hinted at above, if a crawl is causing problems there are likely issues somewhere. Whether this is as simple as straight up server resources or is more due to problems with the site and crawlable URLs needs to be determined but let me give you some pointers.
- SEO Audit - at least a crawl / indexation audit - lets see how many pages we can crawl? How does this stack up against the amount of products / categories? You may well find some easy wins here and sections of the site that can be blocked off or variables you tell Google not to crawl in webmaster tools. Nofollow directives and URLs can be your friend here as well so you tackle it on both fronts.
- Magento Optimisation - it is easy with a system like Magento to create pages that have a heavy burden on the database with hundreds of queries. If these options are not really used (only by crawlers) then they can be audited and removed / improved.
- Server Resources - Magento can be a hungry beast
- Dig into the http access logs to identify who and what is crawling and from where and come up with a list of what you need to block and how.
Summary
Ultimately, blocking all spiders is daft and there is a good chance it won't resolve the issue anyway - that is unless it screws over the clients search visibility so badly that they don't do any traffic! There are likely issues though be that with the site itself or something else so a good way to couch this to them is as their friend and helper - someone who will help them identify and resolve the issues. If it gets combative then it will only be harder to resolve.
Alternatively, you could move to another host. Part of me would suggest doing this anyway as no host should be able to hold you to ransom like this. This one daft move could have potentially ruined the clients visibility in what is a key time of the year for most online businesses. Imagine if they did not have an SEO on board? If they did not have an automated crawl to highlight these issues?
There is certainly a worthwhile exercise here as the site likely has some problems (or at least areas that can be improved upon) so optimisations can be made but, I would still consider jumping ship and moving to an SEO savvy host in the long term if bridges can't be built.
Hope that helps!
MarcusReferences
-
Yeah, what he said...
And when you call, let them know that it was their slow ssa server that caused you to find another host.
-
I agree with your assessment.
This hosting service is being run by either noobs or stingy people or both.
I would get a new host right away. ASAP. Your rankings in search will die completely if you remain on this host.
In addition to what you have seen here they probably have other practices that are deadly.
I would install my site on new server, then change the DNS before informing the current host. Then call to cuss 'em out.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need help related to SEO ?
Hi, Hope you guys are doing well! I need help from experts, I have a query that "Does google allows SEO for websites related to crypto currency ?" If yes, what is best way to do SEO for crypto currency website ?
Technical SEO | | JessicaThompson0 -
Homepage Not Ranking in Google - How long do old (not current) bad SEO practices exert influence?
I'm trying to get to the bottom of a problem I have with the Google ranking for mauiactivities.com - it's far below what I would hope for. My research so far has uncovered the following, and any advice on where to go from here would be appreciated. _Edit:_No problems with Bing or Yahoo - the site is #1 for primary key word 'maui activities' 1. Running a site:http://www.mauiactivities.com search in Google reveals that the homepage doesn't rank. At all. I looked through the 17 pages of results and can't spot it. Edit: I have now, after fresh checks after submitting the homepage through Search Console, found it at #1 - still, the following applies ... 2. I've found that the domain (before it was purchased by my client in 2011) had some bad inbound links, specifically from scubamaui.com (no longer active). The links where white, on a white background. This web archive snapshot will reveal all. 3. Those bad links were 'cleaned up' (i.e. they don't show in the web archive) from 2014, and as mentioned above, the website is now 'down'. 4. Search Console doesn't have a manual penalty. 5. When I search for 'tropical divers maui' in Google I find www.mauiactivities.com is the 4th result. To me, this indicates a current relationship with the dead site (Tropical Divers Maui). No other term comes close to ranking to high for the homepage. So, to summarise - can the old, dead Tropical Divers Maui website still be affecting the Google ranking, and what would you suggest I do next?
Technical SEO | | jsherwin0 -
Subdomain hosted on a different server VS Subfolder on main server
We have a website developed in ColdFusion on a server does not support PHP. We have a blog for the site using WordPress (PHP), hosted on a different server, with a subdomain as the URL. (example: blog.website.com) I've heard that search engines treat subdomains as completely different websites from the main domain, so they could actually be in competition for rankings in the search engines - is that correct? I am under the impression that the traffic to the blog will not show as traffic to the main website, because it is hosted on a different server - is that right? If I am correct, I assume the best solution would be to install PHP on our main server, and put the blog in a subfolder ... or would the subdomain be OK as long as the blog is hosted on the main server? Thanks!
Technical SEO | | vermont0 -
Why are Google search results different if you are log'd into Google or not?
I get different results when I'm log'd into my Google account associated with my website than if I'm not. The same country is occurring. So how can I rely on the google results I'm seeing? For instance my site is page 1 with the improvements I made based on SEOMOZ if I'm log'd in. Yet I'm not on the first 25 pages if I'm not logged in.
Technical SEO | | Romana0 -
Want to Target Mobile site for Google Mobile Version and Desktop Site for Google Desktop Version
I have ecommerce site with both mobile version and desktop version. Mobile version starts with m.example.com and full version starts with www.example.com I am using same content through out both site and using 301 redirection by detecting user agent vice-versa. My both sites are accessible to crawl by any google spider. I have submitted both sites's sitemap to GWT and mobile site having mobile sitemap xml, so google can easily recognize my mobile site. Is it going to help to rank my both sites as per my expectation? I need to rank for mobile site in Google mobile and ranking for desktop site in Google desktop version. Some of pages of my mobile site are started to appearing in Google desktop version. So how I can stop them to appear in Google desktop? Your comments are highly welcome.
Technical SEO | | Hexpress0 -
About Google Spider
Hello, people! I have some questions regarding on Google spider. Many people are saying that "Google spiders only have US IP address." Is this really true? But I also saw video from Google's offical blog and it said "Google spider come from all around the world." At this point I am really confused. Q1) I researched and it seems like Google spiders have only US IP address. THen what does exactly mean by "Google spider come from all around the world."? Q2) If Google spider have only US IP address, what happen to site which use IP delivery? Is this means that Google spider always redirect to us site since they only have US IP? Can anyone help me to understand?? One more questions! When Google analyzing for cloaking issue, do you think Google analyze when spider crawls the site or after they crawled the site?
Technical SEO | | Artience0 -
Help with SEO
Hello, I am brand new to SEO and I'm learning on the go everyday. I am having issues with Google and getting any sort of ranking or analysis or even just traffic reports. I understand the site has never really been optimized so it might really not have any reports. So basically my real question is what helpful tricks or hints do you guys have that I can implement? Anything and everything helps. So far I have run the crawl diagnostics and I'm working on fixing the errors. Thanks fr your help.
Technical SEO | | Future130 -
Google Penalize?
Hello, I read an statement somewhere which stated: "2 identical URLs linked to 2 different popular key phrases next to each other (on the same website/domain) will lead to a Google penalize. Google knows, that both terms are popular. This means, Google will ignore the links to your site (you'll not have any benefit) and the site you have your links on loses authority." What are your thoughts on this statement? Thank you.
Technical SEO | | micfo0