Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
-
Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem.
To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board.
Context that might be helpful:
- I did check technical settings, and the sites are crawlable.
- The sites do appear in search but seem to be losing organic search traffic.
Thanks for any help you can provide!
-
Hey David,
I thought I'd jump in here, as it's our tool
We have more information on bot verification here, including a troubleshooting section with common issues for genuine events being marked as spoofed -
https://www.screamingfrog.co.uk/log-file-analyser/user-guide/configuration/#verify-bots
You can also reach us via our support here - https://www.screamingfrog.co.uk/log-file-analyser/support/
Cheers.
Dan
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Bot Crawling issues
Dear all, Is this cache:www.subhavaastu.com is not working now with Google. Why it is not showing when the site is crawled. What were the new algorithms which Google is adopted? in my searches of all of my site internal links of www.SubhaVaastu.com, I observed only 404 instead of Google visiting time and date. After observing this 404 of each of my site links, I understand Google stopped crawling my site. Some more examples are shown below with other websites: cache:www.vastuwebsite.com ("NOT" showing when Google visited this site) cache:www.vastuconsultantusa.com ("NOT" showing when Google visited this site) cache:www.shubhavaastu.com ("NOT" showing when Google visited this site) cache:www.subhavastu.com (Showing when Google visited this site) To my surprise, I noticed that Google crawled latest links in my site, which I added a new link (https://www.subhavaastu.com/remove-negativity.html) just 10 days back, this new link was clearly crawled by Google. I typed "remove negativity subhavaastu", I saw the results with this new page in the SERP, but on the same way when I typed "cache:www.subhavaastu.com/remove-negativity.html", it is showing again 404. what is happening with Google, is Google is following any new algorithms now. Is google changed any new concept? or, is my site is penalized in any case, I think it may not be, because if my site is penalized, then Google should not visit the new links and should not show the results with my site. Coming with my site, its pure from viruses, no malicious codes. Indeed it's an article based site, which has a good reputation. This domain is taken in the year 2003. We never spam anywhere we never did any wrong methods. If my site is penalized, then is it manual penalized or bot penalized. I thoroughly checked webmaster tools and google console, I never found any notices or any note from Google. Require experts analyzation on this doubt. Thanks in Advance.
Algorithm Updates | | SubhaVaastu0 -
Adding non-important folders to disallow in robots.txt file
Hi all, If we have many non-important folders like /category/ in blog.....these will multiply the links. These are strictly for users who access very rarely but not for bots. Can we add such to disallow list in robots to stop link juice passing from them, so internal linking will me minimised to an extent. Can we add any such paths or pages in disallow list? Is this going to work pure technical or any penalty? Thanks, Satish
Algorithm Updates | | vtmoz0 -
Time taken for Google Algorithm updates to show affect in Middle East?
Hello everyone, Just a quick question. Can anyone give me a safe estimate of how much time it could take for a Google Algorithm Update to show its effect in the Middle East after roll out? Maybe you guys can direct me to a post to read through and learn more about it myself. Your input will be highly appreciated. Regards, Talha
Algorithm Updates | | MTalhaImtiaz0 -
Authorship Photo Not showing in for last 6 months now
Hi - Early last year we activated Authorship Photo which remained in SERP till Nov'12... However, post that the Authorship Photo is not showing, neither the schema rating tags are showing in Search engines We tried a lot by changing lot of hits and tries, still to no avail.. We have written to google twice for it (they had a link for which author whose photo now showing after everything is right can send site links )- but to no avail.... The Rich snippet shows Authorship Photo, Schema rating. Even, Google custom search on our own site own page showing it... However, this does not translated into any of these shown in actual Google Search Engines. Our Site is :- http://www.mycarhelpline.com/ Sample Links of rich snippet http://www.google.com/webmasters/tools/richsnippets?q=http%3A%2F%2Fwww.mycarhelpline.com%2Findex.php%3Foption%3Dcom_easyblog%26view%3Dentry%26id%3D94%26Itemid%3D91&html= http://www.google.com/webmasters/tools/richsnippets?q=http%3A%2F%2Fwww.mycarhelpline.com%2Findex.php%3Foption%3Dcom_latestnews%26view%3Ddetail%26n_id%3D467%26Itemid%3D10&html= Even google custom search showing schema rating tags for searched keywords like :- Ford Ecosport, Tata Nano Diesel .... However, on actual search the schema tags are now shown Can anyone suggest - what am i missing, actually lost on this...... Worse - our SERP are somehow also slowly coming down for some of the main keywords too
Algorithm Updates | | Modi0 -
Does anyone know what it takes to get your Google Plus statuses to show up under the Knowledge Graph?
I've been looking into G+ and how to get the information and status updates in to the Knowledge Graph for small companies and have not been able to. Does anyone know exactly how to do it?
Algorithm Updates | | DragonSearch1 -
Long term plan for a large htaccess file with 301 redirects
We setup a pretty large htaccess file in February for a site that involved over 2,000 lines of 301 redirects from old product url's to new ones. The 'old urls' still get a lot of traffic from product review sites and other pretty good sites which we can't change. We are now trying to reduce the page load times and we're ticking all of the boxes apart from the size of the htaccess file which seems to be causing a considerable hang on load times. The file is currently 410kb big! My question is, what should I do in terms of a long terms strategy and has anyone came across a similar problem? At the moment I am inclined to now remove the 2,000 lines of individual redirects and put in a 'catch all' whereby anything from the old site will go to the new site homepage. Example code: RedirectMatch 301 /acatalog/Manbi_Womens_Ear_Muffs.html /manbi-ear-muffs.html
Algorithm Updates | | gavinhoman
RedirectMatch 301 /acatalog/Manbi_Wrist_Guards.html /manbi-wrist-guards.html There is no consistency between the old urls and the new ones apart from they all sit in the subfolder /acatalog/0 -
Google showing different pages for same search term in uk and usa
Hi Guys, I have an interesting question and think Google is being a bit strange.. Can anyone tell me why when I input the term design agency in Google.co.uk it shows one page, but when i tyupe in the same search term in Google.com (worldwide search) it shows another page.. Any ideas guys? Is this not bit strange?? Any help here be much appreciated.. Thanks Gareth
Algorithm Updates | | GAZ090 -
When Google crawls and indexes a new page does it show up immediately in Google search - "site;"?
We made changes to a site, including the addition of a new page and corresponding link/text changes to existing pages. The changes are not yet showing up in the Google index (“site:”/cache), but, approximately 24 hours after making the changes, The SERP's for this site jumped up. We obtained a new back link about a couple of weeks ago, but it is not yet showing up in OSE, Webmaster Tools, or other tools. Just wondering if you think the Google SERP changes run ahead of what they actually show us in site: or cache updates. Has Google made a significant SERP “adjustment” recently? Thanks.
Algorithm Updates | | richpalpine0