Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
'spammy' domains redirecting to website
Hi Everyone, I hope that someone will be able to help us with this one as we have trawled the internet looking for a solution! We have multiple domains (.com/.co.uk/.net versions) which all point to the one website, however, some of these domains have a high spam score - 9-11. Our first initial reaction would be to remove the auto redirects, but, the other domains have been a source of conversion in previous months (or so analytics tells us). So what I'm wondering, is do we remove the 'spammy' links from redirecting to our site, or do we leave them there? We certainly don't want to risk a penalty. Thanks for reading!
Link Building | | hydra_creative0 -
Blog commenting counter exact match do-follow backlinks?
Hi, I was wondering if blog commenting (no-follow links with names as anchor text) is a good way to counter possible overuse of exact match do-follow backlinks?
Link Building | | anubis20 -
How long does it take Google to give value to a backlink?
Hi Once Google indexes your backlink does it pass value to that link straight away? Before i was under the impression once backlinks were created (providing they were good links) you wouldn't see any value or the serp climb for 6/10 months, is that still the case? Reason i ask is we're link building for a summer item and most if the back links are being indexed within a few days by Google. If the backlinks are given value by Google once indexed then we'll carry on till late summer. If its 6/10 months to be recognized we may start moving onto something else. Advice would be appreciated.
Link Building | | neenan10 -
Which of the following actions caused a drop in keyword ranking?
Hello, About 2 months ago we build a specific page arround the keyword 'kerstkaarten' (christmas cards in Dutch). It moved up to the first page (7th position), but last week we fell back to a +50 position.
Link Building | | ETonnard
The changes we made during the last week where minimal, and I really don't have an idea what may have caused the sudden drop in ranking. Maybe you can help? Link: http://www.smartphoto.be/nl/belangrijke-momenten/kerst/kerstkaarten Changes made: -> I changed the keyword kerstkaarten on tim on the page to 'kerstkaartjes', because the Moz page grader said the keyword density was a bit too high. -> We changed the top image on the page with a different promo. Title of the image changed from 'kerstkaarten-promo' to 'kerstkaarten-wenskaarten' -> The date on a small legal text on the page changed -> We got 2 backlinks from bloggers who've written an article about our christmas cards. So no major changes in my opinion. Any ideas? Thanks a lot! hz2DpJS0 -
I have just read an article that says no follows now leak link juice?
I have just been reading a really interesting article about link juice. The way I read the piece the author was saying that there has been a change in the way link juice is blocked with 'no follows.' The author seemed to say that there has been a change and now even if you add a 'no follow' onto a page that the page will still send link juice to that page. I may have read this wrong. I just wanted to check if anyone else had heard this at all? Many thanks
Link Building | | mblsolutions0 -
I'm thinking about buying a competitor and 301 redirecting? How much SEO value?
I'm thinking about buying a competitor and 301 redirecting their site to mine. The high level stats are as follows. My site has a DA of 46 and the homepage has a PA of 55. 375 root domains, 170,000 links. The site sells millions of dollars worth of product each year. The competitor (who I had never heard of) has a DA of 58 and a homepage PA of 64. They have 634 root domains and 260,000 links. They aren't selling much of anything (less than 100,000 per year). We might be able to operate their site but I'm concerned about maintaining 2 platforms. My question is about the value of buying this site and 301 redirecting it to my site. Would this create long term SEO value or not? Any examples that have been documented are greatly appreciated.
Link Building | | bradwayland0 -
Does Google see this as an inbound link?
Hi, I'm looking at adding my site to a directory. It's highly relevant to my industry and has relatively good PA/DA. The external links on the directory uses some PHP code like the following: http://www.ThisDirectory.com/visit.php?i=3141635&url=http://www.ExternalDomain.com Obviously the links works as a redirect but they do not show up in OSE. Would Google count this link as an inbound link? Thanks,
Link Building | | Confetti_Wedding0 -
Google dosent care that much about links lately?
Google dosent care that much about links lately or crawl started beeing more intelegent about it? Do you feel any changes?
Link Building | | DiamondJewelryEmpire0