Help Blocking Crawlers. Huge Spike in "Direct Visits" with 96% Bounce Rate & Low Pages/Visit.
-
Hello,
I'm hoping one of you search geniuses can help me.
We have a successful client who started seeing a HUGE spike in direct visits as reported by Google Analytics. This traffic now represents approximately 70% of all website traffic. These "direct visits" have a bounce rate of 96%+ and only 1-2 pages/visit. This is skewing our analytics in a big way and rendering them pretty much useless. I suspect this is some sort of crawler activity but we have no access to the server log files to verify this or identify the culprit. The client's site is on a GoDaddy Managed WordPress hosting account.
The way I see it, there are a couple of possibilities.
1.) Our client's competitors are scraping the site on a regular basis to stay on top of site modifications, keyword emphasis, etc. It seems like whenever we make meaningful changes to the site, one of their competitors does a knock-off a few days later. Hmmm.2.) Our client's competitors have this crawler hitting the site thousands of times a day to raise bounce rates and decrease the average time on site, which could like have an negative impact on SEO. Correct me if I'm wrong but I don't believe Google is going to reward sites with 90% bounce rates, 1-2 pages/visit and an 18 second average time on site.
The bottom line is that we need to identify these bogus "direct visits" and find a way to block them. I've seen several WordPress plugins that claim to help with this but I certainly don't want to block valid crawlers, especially Google, from accessing the site.
If someone out there could please weigh in on this and help us resolve the issue, I'd really appreciate it. Heck, I'll even name my third-born after you.
Thanks for your help.
Eric
-
Hi SirMax,
Thanks for your input. I appreciate it. We'll add Wordfence to our WordPress toolbox and see if that addresses the issue.
In response to previous posts, thanks to everyone for your input. We were able to apply some filters to remove the bogus bot traffic from the analytics and normalize the data, however, this did not actually resolve the issue and in my eyes is more of a BandAid fix. The evil crawlers are still there, we just can't see them.
Thanks again for all of your input.
Eric
-
Hostname filtering does not work any more. Unfortunately most of the spammers have adapted and are using your website as hostname.
For the WordPress I use Wordfence plugin( using paid version - not affiliated with them in any shape or form beyond paying for their services). In the advance blocking you can set limits on how fast and how many pages crawlers can request. You can also block by country or ip range. It can also show you live traffic with a lot of details ( a lot more then google analytic - more like server log ). It might not be the complete remedy but it can help.
-
I wish I had an answer for how to stop the bots from hitting your site at all - I don't think a good one exists, as any solutions that wouldn't also block real human traffic to your site are going to be easy for spam bots to get around. I think your best bet is just to do everything you can to keep your data as clean as possible.
-
Hi Ruth,
Thanks a bunch for taking the time to respond to my post. Great advice. This is reassuring on a number of levels, however, it doesn't address the underlying issue of how to stop these spam bots in the first place.
We've already started the process of filtering out some of this bogus data. We'll also be integrating some WordPress plugins to see if that helps. That said, if the spam bots are hitting Analytics directly, as opposed to the actual website, WP plugins won't do anything.
Anyway, I appreciate your input and advice. Thanks so much.
Eric
-
Hi Eric,
A few things to reassure you off the bat:
- For what it's worth, there is a huge, HUGE amount of crawler spam happening in the web today. Every site I work on is being hit hard with false referrals and direct visits. I know Google Analytics is working on a solution to better filter these visits out. So I wouldn't be too concerned that it is something a competitor is doing to your site, specifically - it's more likely that it's been caught up in the general wave of spam crawlers.
- It's important to note that when we talk about Google looking at bounce rate and dwell time as part of ranking your site, those numbers are specifically from clicks through from search - that's data that Google can get without using your private web analytics data as a ranking factor, which they've said repeatedly that they don't and won't do. So a bunch of direct visits with high bounce rates will NOT affect your rankings.
So, it's not dangerous, just annoying. On to how to get that data out of your reports:
- Make sure you're not filtering out spam referrers at a View level - this can cause those visits to incorrectly appear as direct traffic.
- You could set up an Advanced Segment in Google Analytics to filter out direct visits with visit times of, say, under 5 seconds. Some real traffic may get caught in that, but it will get the noise levels down.
- The best way to filter out spam bot traffic, in my opinion, is to set up hostname filtering. Here's a post on Megalytic on how to do that: https://megalytic.com/blog/how-to-filter-out-fake-referrals-and-other-google-analytics-spam. Make sure you've also got an "Unfiltered Data" View so you'll still have historic raw data if you need it.
Hope that helps! Good luck.
-
Check webserver log files, or log visits (ip address, user agent, __utma, __utmz, possibly browser fingerprint, etc...)
Analyzing those you can easily find out if the traffic is from scraping bot or humans.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why might my websites crawl rate....explode?
Hi Mozzers, I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically: 9/5/16 - 923
Reporting & Analytics | | Silkstream
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809 I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice.1 -
Tool to check page size for multiple url's at once
In Google Analytics under Site Speed > Page Timings, you can see all pages and their loading time compared to the average. This is very handy to check which pages maybe need some optimization. I would also like to check the size for these pages in a similar way. There are multiple tools out there like GTmetix and Pingdom that give specific information and performance insights. The problem is that they are limited to check one url at a time. Does someone know about a tool to check the page size of multiple url’s at once (and if possible to easily export to Excel)? That way I can check which pages are big in size and research/optimize them. Thanks in advance
Reporting & Analytics | | Mark.0 -
Can't seem to rank for keyword "home care grand rapids" - need some advice
I am trying to rank for "home care grand rapids" and am having a really difficult time. My site: http://healthcareassociates.net has better backlinks, keywords and other seo markers than my competitors but I still can't seem to rank. The keyword and associated keywords (home care grand rapids michigan, home health care grand rapids, etc.) are only 31-33% difficulty and my site/page rank is better than the leading sites. What gives? Todd
Reporting & Analytics | | t1kuslik0 -
Why is Google Analytics showing index.php after every page URL?
Hi, My client's site has GA tracking code gathering correct data on the site, but the pages are listed in GA as having /index.php at the end of every URL, although this does not appear when you visit the site pages. Even if there is a redirect happening for site visitors, shouldn't GA be showing the pages as their redirect destination, i.e. the URL that visitors actually see? Could this discrepancy be adversely affecting my search performance? Example page: http://freshstarttax.com/innocent-spouse/ shows up in GA as http://freshstarttax.com/innocent-spouse/index.php thanks
Reporting & Analytics | | JMagary0 -
Why did my home page fall off of google rankings?
My home page at www.smt-associates.com has been ranked well for various key word phrases for years. I've tried to optimize it for the search "Crystal Lake CPA Firm" and it always had ranked number 1-2. Now it doesn't even rank in the top 5 pages (actually I don't know which page it falls on). I did an on-page report card and it has an A rating. So, what is preventing Google from ranking my home page on page 1? There's not that much competition so this should be an easy ranking for me. I don't know how ling this has not been listed, but I did modify my site about 12-18 months ago with a new WP theme. Could the theme be the problem?
Reporting & Analytics | | smtcpa0 -
Has anyone experienced Google Analytics track the page visit to a "thank you" page, but not the goal conversion?
Has anyone experienced where Google Analytics would track the page visit to a "thank you" page, but not the goal conversion that should result? The goal had worked for a long time as it is as just a goal url with head match. No funnel. Not case sensitive. For about four days now, no conversions have been recorded, but Google Analytics shows hundreds of people visited the page that should trigger the goal. Additionally, we have received the hundreds of leads. A Screaming Frog search shows the code is embedded throughout the site. For the interested, the GA code looks like (and the 8 Xs are the correct number on the site): Am I missing something?
Reporting & Analytics | | 352inc0 -
Why is my direct traffic down DRASTICALLY?
I have been seeing a trend for a while that is intesifying. My direct traffic numbers are down A LOT. We are not down 50% to LY (in actual number not just percentage of traffic) I am trying to understand what could be the causes of this issue. I was considering simply bigger meaner competition, but I actually perform decently on my returning customers. Also my performance on my brand keyword is more inline with my current trend so I would except these KW to do equally as bad if the actual brand/store was the issue. The more surprising even, is the fact that I can trace back the start of the trend exactly to the day. Overnight on Sept 22 LY direct traffic went down 30% (to LY) when it was trending UP 20-25%(to LY) before. Now, we did do a redesign of the website on May 2011 (4 months before the drop), and did change host Oct 2011 (a couple weeks after the start of the trend). Do you have any clue as to why this could be happening? Did GA start tracking direct traffic differently?
Reporting & Analytics | | CassisGroup
Any thoughts?0 -
Stats show /blog/wp-cron.php at the top. What is it?
Hi, I have worked with websites for years but have no clue when it comes to Wordpress. We have our main website and then a Wordpress blog running in a subfolder that is only about a year old. The blog has only 7 posts so you can see how small it is vs main website with 200 pages. Usually our main index page of the site is at the top of the stats with the most views and this page /blog/wp-cron.php is about 30% lower. Now suddenly over the last month this page has jumped to the top and accessed almost as much as the home page of the site. We took a big hit with the latest Google Update so we are tyring to determine if there is anything technical in our site that has caused an issue. Thanks in advance Force7
Reporting & Analytics | | Force70