Recovering from index problem (Take two)
-
Hi all. This is my second pass at the problem. Thank you for your responses before, I think I'm narrowing it down!
Below is my original message. Afterwards, I've added some update info.
For a while, we've been working on http://thewilddeckcompany.co.uk/. Everything was going swimmingly, and we had a top 5 ranking for the term 'bird hides' for this page - http://thewilddeckcompany.co.uk/products/bird-hides.
Then disaster struck! The client added a link with a faulty parameter in the Joomla back end that caused a bunch of duplicate content issues. Before this happened, all the site's 19 pages were indexed. Now it's just a handful, including the faulty URL (thewilddeckcompany.co.uk/index.php?id=13)
This shows the issue pretty clearly.
I've removed the link, redirected the bad URL, updated the site map and got some new links pointing at the site to resolve the problem. Yet almost two month later, the bad URL is still showing in the SERPs and the indexing problem is still there.
UPDATE
OK, since then I've blocked the faulty parameter in the robots.txt file. Now that page has disappeared, but the right one - http://thewilddeckcompany.co.uk/products/bird-hides - has not been indexed. It's been like this for several week.
Any ideas would be much appreciated!
-
Thank you all, this is brilliant.
-
Your problem is with the robots.txt file. You are blocking the URL
thewilddeckcompany.co.uk/index.php?id=13
That URL 301 redirects to the correct URL of
http://thewilddeckcompany.co.uk/products/bird-hides
Google cannot "see" the 301 redirect from the old "bad" URLs to the new "good" URL.
You have to let Google crawl the old URLs and see the 301 redirects so that it knows how things need to forward.
I would do this for all the duplicate pages, make sure they 301 to the correct pages and do not put the "bad" pages in robots.txt - otherwise the indexing will not be updated.
Something separate to check. We have seen Google taking a while to acknowledge some of our 301s. Go into your GWT and look at your duplicate title reports. You may see the old and new URLs showing as duplicates, even with the 301s in place. We had to setup a self canonicalizing link on the "good" pages to help get that cleaned up.
-
Blink-SEO
Jonathan is correct to try a Fetch as Google in WMT for the urls you need re indexed. (Note, that is not really the purpose of a Fetch as Google, but sometimes it works.)
I would also resubmit the sitemap now that you have blocked the offending url with robots.txt. It is likely the resubmission will help you the quickest IMO.Best,
Robert
-
It sounds like you just need to wait for Google to recrawl your robots.txt file. I saw this error in the serps:
www.thewilddeckcompany.co.uk/products/timber-water...
A description for this result is not available because of this site's robots.txt – learn more.So it is clear that the robots.txt file has not updated with the changes, after the mistake was made. Try fetching as Googlebot within webmaster tools, but it may take a little time to update. But at least it would seem that the robots.txt error is still a cause of the problem, just need to wait a little longer.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Optimising for two countries
I have a UK-based client, with a global product, who wants his website optimised for the same keywords in Google US and Google UK. Should I treat this as two separate jobs in any way or does it make no difference? I'm particularly wondering about link building and onsite optimisation
Intermediate & Advanced SEO | | markwes0 -
Old pages STILL indexed...
Our new website has been live for around 3 months and the URL structure has completely changed. We weren't able to dynamically create 301 redirects for over 5,000 of our products because of how different the URL's were so we've been redirecting them as and when. 3 months on and we're still getting hundreds of 404 errors daily in our Webmaster Tools account. I've checked the server logs and it looks like Bing Bot still seems to want to crawl our old /product/ URL's. Also, if I perform a "site:example.co.uk/product" on Google or Bing - lots of results are still returned, indicating the both still haven't dropped them from their index. Should I ignore the 404 errors and continue to wait for them to drop off or should I just block /product/ in my robots.txt? After 3 months I'd have thought they'd have naturally dropped off by now! I'm half-debating this: User-agent: *
Intermediate & Advanced SEO | | LiamMcArthur
Disallow: /some-directory-for-all/* User-agent: Bingbot
User-agent: MSNBot
Disallow: /product/ Sitemap: http://www.example.co.uk/sitemap.xml0 -
A Tale of Two (Competing) Domains
Bit of a weird one but I'm hoping someone can help our team of two sort it out. I'm a copywriter/marketer who's been learning SEO on-the-go, along with our web developer, for the past ~9 months. We've inherited what I consider to be a mess of a situation involving two main e-commerce sites. The company has a sporadic history of spinning off brands, in hopes to either build business in a new market or sell the brands off or what-have-you. The original company rose to prominence manufacturing disc packaging and selling it to software companies, which has obviously been on the way out for some time now. So they've dipped into a handful of other business products for marketing/office use. The company used to sell all products, in individual AND box quantity, on one site (since 1996). In 2012 they decided to move individual quantity sales to a different site with the domain name of one of the brands, focus it more on consumers and small business, etc. We have more flexibility to make changes to the consumer site, so in my opinion it's in better shape. The consumer site (DA 39) offers "retail pricing" with flat rate shipping and free shipping over $25. The b2b site (DA 37) offers "industry pricing" with a weight-based shipping model. Traffic on the business site is down 70% since 2010. We've also been asked to take certain products down in hopes that viewers will pick up the phone and buy a customized version from a sales rep instead. Since probably 3/4 of the products are on both sites, nearly all the category and product pages are competing in SERPs. Not only that, but the business site's product pages invariably link to the corresponding page on the consumer site -- hundreds of links pointing to the consumer site. We know for a fact that people are price checking product+shipping between our own two sites. The issues are further exacerbated because we have even more spinoff domains -- an informative site for a particularly successful product line, an e-commerce site just for vinyl products, etc. etc. So I guess I'm trying to figure out how to make the most of the situation we're now in. Our hands are somewhat tied because we're not 'decision makers'. But we've got a meeting tomorrow to talk about the future of one of the sites, so I figure I at least want to be informed. I am concerned about making further decisions without considering the consequences, especially when our bonuses are tied to web sales... I feel like this is just scratching the surface of the problem so let me know if you guys have further questions.
Intermediate & Advanced SEO | | UEUP0 -
Keyword rich internal linking - problem?
Had an interesting situation today.. We write daily news articles on our site. In each article we link out to two sources that we are writing about (credible sources) and we do one or two internal links. For example.. 'Today McDonald's have announced that they are purchasing more blue widgets in order to increase their opportunity to appeal to a larger market.' So in that sentence you can see one outbound link and one inbound to blue widgets on our site. I got an email today from a large company who we have written an article about in the industry and they have asked me to remove the link to their site.. I actually asked them why and this was their response. 'We're concerned because of the number of keyword-rich internal links in the article, and are worried that being included alongside them might be misinterpreted by Google as an artificial link.' Fristly, do they really have anything to be worried about?.. but more importantly, with our internal linking, do we have anything to be worried about?.
Intermediate & Advanced SEO | | nick-name1230 -
Site Search Results in Index -- Help
Hi, I made a mistake on my site, long story short, I have a bunch of search results page in the Google index. (I made a navigation page full of common search terms, and made internal links to a respective search results page for each common search term.) Google crawled the site, saw the links and now those search results pages are indexed. I made versions of the indexed search results pages into proper category pages with good URLs and am ready to go live/ replace the pages and links. But, I am a little unsure how to do it /what the effects can be: Will there be duplicate content issues if I just replace the bad, search results links/URLs with the good, category page links/URLs on the navi. page? (is a short term risk worth it?) Should I get the search results pages de-indexed first and then relaunch the navi. page with the correct category URLs? Should I do a robots.txt disallow directive for search results? Should I use Google's URL removal tool to remove those indexed search results pages for a quick fix, or will this cause more harm than good? Time is not the biggest issue, I want to do it right, because those indexed search results pages do attract traffic and the navi. page has been great for usability. Any suggestions would be great. I have been reading a ton on this topic, but maybe someone can give me more specific advice. Thanks in advance, hopefully this all makes sense.
Intermediate & Advanced SEO | | IOSC1 -
How to Find problem domain history
Hi I have what most of you may think is a dumb question but here goes. please be nice... 🙂 So I have a client (http://www,ace-alarms.co.uk) who are having a real problem ranking for ANY of their key words. I know it's a reasonably competitive area but I've not seen such a stubborn domain and it seems that no matter what we do there's nothing listed. i'm thinking that there may be a problem with the domain name. My question is; how can I find out if this is a problem domain. Thanks in advance Steve
Intermediate & Advanced SEO | | stevecounsell0 -
Ajax Content Indexed
I used the following guide to implement the endless scroll https://developers.google.com/webmasters/ajax-crawling/docs/getting-started crawlers and correctly reads all URLs the command "site:" show me all indexed Url with #!key=value I want it to be indexed only the first URL, for the other Urls I would be scanned but not indexed like if there were the robots meta tag "noindex, follow" how I can do?
Intermediate & Advanced SEO | | wwmind1 -
Bridge page problem
Hello I run this website, http://www.bestillkredittkort.no (Norwegian website) I'm working all i can to make it rank, so i wanted to test adwords to see how good the page converted.
Intermediate & Advanced SEO | | katal
.
Problem is my page got labeled bridge page. I have read the Google guideline for fixing bridge page and tried to fix what they suggest to make them accept the page, but its not working. I might think there's a underlying problem here, but im not sure how to fix it. I've even seen People from the same niche running adwords campaign with less content on the target page. Last time i tried to recheck if my site would get approved. I ran over the quality score tab. And when it was in pending it showed 10/10 in every aspect. was that just a sample ? I'm realy confused on this one. And im afraid it might be a problem witht the page that can destroy my seo efforts. Anyone have any suggestion or feedback on this one?0