Is Googlebot ignoring directives? Or is it Me?
-
I saw an answer to a question in this forum a few days ago, that said it was a bad idea to use robots.txt to tell googlebot to go away.
That SEO said it was much better to use the META tag to say noindex,nofollow.
So I removed the robots directive and added the META tag
<meta robots='noindex,nofollow'>
Today, I see google showing my send to a friend page where I expected the real page to be.
Does it mean Google is stupid?
Does it mean google ignores the Robots META tag?
Does it mean short pages have more value than long pages?
Does it mean if I convert my whole site to snippets, I'll get more traffic?
Does it mean garbage trumps content?
I have more questions, but this is more than enough.
-
Thank you Ryan.
They completely ignored the meta tags., completely messing up our serps. So I put it back in robots. I wont trust google again to do the right thing.
-
Hi Allan,
It is a best practice to use meta tags to indicate your indexing preference to search engines.
Normally the recommended implementation would be "noindex, follow" but without examining your site it is impossible to know for sure.
Google honors meta tags but there are a number of issues which could be the source of your issue. For example, if you did not use valid syntax the tag may not be honored. If you are blocking the page in robots.txt, then search engines cannot read the tag.
As for the last three questions, the simple answer is quality content is best.
If you can share the URL of the page involved, we can offer a specific response to the implementation of the meta tag.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My posts are ignored by google not sure why
I have been investigating this problem for some time now and there must be a technical problem. My posts seem to be ignored by google. For example, if i put this title of my article in google, other sites come up instead of my website Lincolnshire Caravan Owners Struggle To Compete With Haven https://www.in2town.co.uk/skegness-news/lincolnshire-caravan-owners-struggle-to-compete-with-haven/ I am trying to find out what technical problem I have that is stopping google displaying my post. Can anyone advise me on what tools to use and how to find out what is going wrong
Technical SEO | | headlinesplus0 -
Spam: how to filter out direct website visitors from Ukraine and Germany
Hi there, I was reading this article: https://mza.seotoolninja.com/blog/stop-ghost-spam-in-google-analytics-with-one-filter as I am experiencing a lot of visitors from Ukraine and Germany as direct website visitors with only 1 pageview, a huge bounce rate (of course) and little time spend, so I am convinced its spam. In the article, it explains how to find the hostname to set up a filter but if I do so, I see that the highest amount of website visitors with this behavior come from the hostname that has my own domain name in it, so I can't exclude this. What should I do?
Technical SEO | | MarketingRumors0 -
Why Can't Googlebot Fetch Its Own Map on Our Site?
I created a custom map using google maps creator and I embedded it on our site. However, when I ran the fetch and render through Search Console, it said it was blocked by our robots.txt file. I read in the Search Console Help section that: 'For resources blocked by robots.txt files that you don't own, reach out to the resource site owners and ask them to unblock those resources to Googlebot." I did not setup our robtos.txt file. However, I can't imagine it would be setup to block google from crawling a map. i will look into that, but before I go messing with it (since I'm not familiar with it) does google automatically block their maps from their own googlebot? Has anyone encountered this before? Here is what the robot.txt file says in Search Console: User-agent: * Allow: /maps/api/js? Allow: /maps/api/js/DirectionsService.Route Allow: /maps/api/js/DistanceMatrixService.GetDistanceMatrix Allow: /maps/api/js/ElevationService.GetElevationForLine Allow: /maps/api/js/GeocodeService.Search Allow: /maps/api/js/KmlOverlayService.GetFeature Allow: /maps/api/js/KmlOverlayService.GetOverlays Allow: /maps/api/js/LayersService.GetFeature Disallow: / Any assistance would be greatly appreciated. Thanks, Ruben
Technical SEO | | KempRugeLawGroup1 -
How are Server side redirects perceived compared to direct links (on a Directory site)
Hi, Im creating some listings for a client on a relevant b2b directory (a good quality directory) I asked if the links are 'followed' or no 'followed' and they said they are 'server side redirects' so no direct links. Does anyone know how these are likely to be perceived by Google ? All BEst Dan
Technical SEO | | Dan-Lawrence1 -
Google showing https:// page in search results but directing to http:// page
We're a bit confused as to why Google shows a secure page https:// URL in the results for some of our pages. This includes our homepage. But when you click through it isn't taking you to the https:// page, just the normal unsecured page. This isn't happening for all of our results, most of our deeper content results are not showing as https://. I thought this might have something to do with Google conducting searches behind secure pages now, but this problem doesn't seem to affect other sites and our competitors. Any ideas as to why this is happening and how we get around it?
Technical SEO | | amiraicaew0 -
Will a google map loaded "on scroll" be ignored by the crawler?
One of my pages has two Google maps on it. This leads to a fairly high keyword density for words like "data", "map data" etc. Since one of the maps is basically at the bottom of the page I thought of loading it "on scroll" as soon as its container becomes visible (before loading the map div should be empty). Will the map then still be craweld by google (can they execute the JS in a way that the map is loaded anyways?) or would this help to reduce the keywords introduced by the maps?
Technical SEO | | ddspg0 -
Is there a penalty for too many 301 re-directs?
We are thinking of restructuring the URLs on our site and I was wondering if there is a penalty associated with setting up so many 301 re-directs.
Technical SEO | | nicole.healthline0 -
Is it possibly to use anything besides a 302 re-direct when your doing a re-direct for someone to login?
Hopefully this makes sense. So I am working on a site that uses a 302 re-direct for logins. As in it goes from a profile page to the login via a re-direct, most of the time I see sites use this as a meta refresh, but in this case I wasn't sure. Obviously when I run a crawl diagnostic I'm getting a lot of errors as in over 100. Now I know there is no link juice with this, but I was just wondering what other people thought on using 302's for logins? Thanks
Technical SEO | | kateG12980