Homepage not indexed - seems to defy explanation
-
Hey folks
Hoping to get some more eyes on a specific problem I am seeing with a clients site.
Site: http:www.ukjuicers.com
We have checked everything we can think of and the usual suspects here are not present:
- Canonical URL is in place
- Site is shown as indexed in search console
- No Crawl, DNS, Connectivity or server errors
- No robots.txt blocking - verified in search console
- No robots meta tags or directives
- Fetch as Google works
- Fetch & render works
- site command returns all other pages
- info command does not return the homepage
- homepage is cached and cache has been updated since this issue started: http://webcache.googleusercontent.com/search?q=cache:www.ukjuicers.com
- homepage is indexed in yahoo and Bing
- all variations redirect to the www.ukjuicers.com domain (.co.uk, .com, www, sans www etc)
The only issue I found after some extensive digging was some issues with the HTTP and HTTPS versions of the site both being available and both specifying the canonical version as themselves. So, http site used canonicals with http and https site used canonicals with https. So, a conflict there with the canonical exacerbating the problem it is there to solve.
The HTTPS site is not indexed though and we have set this up in webmaster tools and now the web developer has set redirects to ensure all versions even the https now 301 redirect to the http://www.ukjuicers.com page so these canonical issues have been ironed out.
But... it's still not indexing the homepage.
The practical implications of this are quite scary - the site used to be somewhere between 1st and 4th for keywords like 'juicers', 'juicer' etc. Now they are bottom of page 1 or top of page 2 with an internal page. They were jostling with the big boys (amazon, argos, john lewis etc) but now they are right at the bottom of the second page.
It's a strange one - i have seen all manor of technical problems over the years but this one seems to defy sensible explanation. The next step is to do a full technical SEO audit of the site but I am always of the opinion that with many eyes all bugs are shallow so if anyone has any input or experience with odd indexation problems like this would love to get your input.
Cheers
Marcus -
Glad you figured it out. I honestly didn't think it would have been the canonicals. I'm a little surprised that the bots didn't just choose not to respect the suggestion as opposed to blanking your site from the index. Didn't think that was even a possibility from incorrect canonicals. Good to know for the future though in case anything like this comes up with anyone else's site.
-
Yep - it's back. Looks like resolving the canonical issue fixed it. Seems it was a usual suspect after all.
-
Yep - bit of a weird one but in the end looks like the canonicals were the issue. Thanks for taking a look though man - super appreciated.
-
Hey Bernadette - thanks for the feedback. Site is back in the index now, looks like the canonicals were the culprit but the owners are keen for no future issues so I will dig in and take a look at these points. Cheers!
-
Hey folks
24 hours after we identified and fixed the canonical issue the site is now indexed again so it does look like it was indeed a canonical conundrum. Both the HTTP and HTTPS sites were claiming to be the canonical version so in some respects creating a conflict. We removed this conflict and it is now indexed.
Thanks for the extra eyes folks - appreciated and if anyone ever needs another pair of eyes to look a problem give me a shout.
Cheers
Marcus -
Hey Marcus. You just need some links from high authority website like moz:) People say you're indexed so case closed, job done:)
-
I just noticed that clicking on the entire slider, even out to the sides where it appears to be just white space, takes you to another page. At first I didn't realize what I was clicking that got me to the next page. When I do Crtl+A on the page, the full width of the slider images shows highlighted in blue, but to the side of those images outside of those bounds is linked. I'm wondering if Google sees this as cloaking and kicked out the homepage as a result.
*I did see that AGM pointed out it's indexed now, but that's not to say this wasn't the cause of original de-index.
-
As of this writing it looks like the page is indexed. By searching site:ukjuicers.com it comes up in the search results with about 861 other results. Not sure if there is anything you changed to get things working again but it seems to be in their index now.
-
I took a look at all of the usual suspects as well... which amounts to pretty much everything that everyone else mentioned but I was intrigued by this issue and thought maybe another set of eyes might notice something that was off. Nothing was wrong in the page source from what I saw, no issues crawling it myself and I didn't see any penalties. Normally I'd think that if your homepage wasn't appearing for branded organic searches then a penalty was levied against you but when that is the case the homepage is still normally find-able in a Site operator search. M__aybe it is related to all the backlinks that were lost/deleted in the past month but I'm not sure why that would be the case unless removing the homepage from the index was a Penguin response to link issues... but I was under the impression that peguin was devaluing the link source not the link recipient and deleting/removing links seems to be a preferred method of handling penguin-related issues. So if there is a relationship between penguin and your homepage being deindexed then I am not sure at all why nor am I certain how to fix it as I'm not seeing anything in particular that screams "linking issue" at me. (though I only did a fairly cursory inspection of things)
So I am stumped. Whenever the issue is figure out I would love to know how/why this came to be.
-
Marcus, I know this is frustrating. I've checked several things, and looked at many of the possibilities that you've already brought up. I don't have access to the Google Search Console, so I cannot comment about any of that data. I'm assuming that you don't have a manual action on the site or any other messages from Google.
What I've seen in the past is issues with schema markup, especially when it comes to reviews and how they're handled on sites. I'm not saying that this is the issue--but I've seen issues that Google has had with these (especially because there is the word "hidden" there in the code). So, you might look into that some more.
The issue could also be related to links--look at the links to the site's home page to see if there is an issue with low quality links pointing to that page or other unnatural links.
If someone has copied the page, added a canonical tag, and then added a "meta noindex tag" to their page, it's possible that they could have taken your page out of the index. This has happened before.
-
Unfortunately you're not amazon so maybe you must try harder;)
or force to index mainpage with some software or indexer website then wait a while.
I'm thinking about some negative seo made for your mainpage but so far can't see any symptoms.
-
This is a strange one then.... very strange.
Just performed a site: search and like you said it is not showing up as indexed. There is normally something technical to explain an issue like this, but I cannot see anything after looking at your site robots and source code.
-
Hey Krzysztof
Yeah, the page has little textual content but... neither does the amazon homepage. Ultimately the page is a jump in point for all the products and the content suits that. Certainly, I could understand Google not liking the page but would that not result in a reduced rank rather than a complete removal like this?
On the dodgy links front they have never done anything on that front - so anything there would be surprising (or just incidental cruft that is out there on scraper sites and the like).
Super odd.
-
Yep - super odd. 15 years or so in this game and never seen anything quite like this. Transient drops but usually it boiled down to some simple technical error or more often user error cough no index / robots.txt cough
-
Hey - the real issue here is the page is just not indexed. It's not there. Not that another page is a more suitable or preferential result. Ultimately that was the best page for a user to jump in at... The page is not even returned in a brand search so... can't see how any other page could be more suitable for that kind of search.
-
Hi Marcus
The only thing I think it can be the issue is the number of words on mainpage. Mostly I see images and words from menus, links and not main content. Digging deeper can help (seo audit).
This can be a penguin too but to know the answer, full link analysis is needed. After quick glance I see some unnatural links but not in larger number. Maybe they got footprints not visible at once (same ip, c class, content with link etc).
-
You're not kidding, this does defy explanation. When did it drop out of the index?
In all honesty, I don't have a solution, you've already checked everything I would have. I'm mostly commenting so I can keep up with this issue and see how it unfolds. Very curious to see if anyone can identify what's happening here.
-
Hmmm, is it a case of Google simply feels the homepage is not as engaging and relevant in terms of search to your users and they put more emphasis on product pages which it choose to feature instead.
I often find that for key terms our product pages almost always rank higher then the homepage unless a brand only search.
Secondly, is this a recent change? Could the most recent Penguin update have simply resulted in your competitors getting a boost where as before the previous algo was holding them back which has resulted in your position slide.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can't get Google to index our site although all seems very good
Hi there, I am having issues getting our new site, https://vintners.co indexed by Google although it seems all technical and content requirements are well in place for it. In the past, I had way poorer websites running with very bad setups and performance indexed faster. What's concerning me, among others, is that the crawler of Google comes from time to time when looking on Google Search Console but does not seem to make progress or to even follow any link and the evolution does not seem to do what google says in GSC help. For instance, our sitemap.xml was submitted, for a few days, it seemed like it had an impact as many pages were then visible in the coverage report, showing them as "detected but not yet indexed" and now, they disappeared from the coverage report, it's like if it was not detected any more. Anybody has any advice to speed up or accelerate the indexing of a new website like ours? It's been launched since now almost two months and I was expected, at least on some core keywords, to quickly get indexed.
Technical SEO | | rolandvintners1 -
Indexed, but not shown in search result
Hi all We face this problem for www.residentiebosrand.be, which is well programmed, added to Google Search Console and indexed. Web pages are shown in Google for site:www.residentiebosrand.be. Website has been online for 7 weeks, but still no search results. Could you guys look at the update below? Thanks!
Technical SEO | | conversal0 -
Follow no-index
I have a question about the right way to not index pages: With a canonical or follow no-index. First we have a blog page: **Blogpage **
Technical SEO | | Happy-SEO
URL: /blog/
index follow Page 2 blog:
URL: /blog?=p2
index follow
rel="prev" /blog/
el="next" ?=p3 Nothing strange here i guess. But we also have other pages with chance on duplicate content: /SEO-category/
/SEO-category/view-more/ Because i don't want the "view-more" items to be indexed i want to set it on: follow no-index (follow to reach pages). But now the "view-more" also have pagination. What is the best way? Option 1:
/SEO-category/view-more/
Follow no-index /SEO-category/view-more?=p2
Follow no-index
rel="prev" /view-more/
el="next" ?=p3 Option 2: /SEO-category/view-more/
Canonical: /SEO-category/ /SEO-category/view-more?=p2
rel="prev" /view-more/
el="next" ?=p3 Option 3: Other suggests? Thanks!0 -
Sitemap url's not being indexed
There is an issue on one of our sites regarding many of the sitemap url's not being indexed. (at least 70% is not being indexed) The url's in the sitemap are normal url's without any strange characters attached to them, but after looking into it, it seems a lot of the url's get a #. + a number sequence attached to them once you actually go to that url. We are not sure if the "addthis" bookmark could cause this, or if it's another script doing it. For example Url in the sitemap: http://example.com/example-category/0246 Url once you actually go to that link: http://example.com/example-category/0246#.VR5a Just for further information, the XML file does not have any style information associated with it and is in it's most basic form. Has anyone had similar issues with their sitemap not being indexed properly ?...Could this be the cause of many of these url's not being indexed ? Thanks all for your help.
Technical SEO | | GreenStone0 -
"Site:" without Homepage, Why?
Hi all, When I put "site:bettingexchange.it" on www.google.it in the SERP it's NOT showed the HOMEPAGE "bettingexchange.it". Google starts with other pages lik "bettingexchange.it/siti/". It's the first time I see something like this, How is it possibile?
Technical SEO | | bettingexchange
How can I reintroduce the homepage?0 -
Getting Pages Indexed That Are Not In The Main Navigation
Hi All, Hoping you can help me out with a couple of questions I have. I am looking to create SEO friendly landing pages optimized for long tail keywords to increase site traffic and conversions. These pages will not live on the main navigation. I am wondering what the best way to get these pages indexed is? Internal text linking, adding to the sitemap? What have you done in this situation? I know that these pages cannot be orphaned pages and they need to be linked to somewhere. Looking for some tips to do this properly and to ensure that they can become indexed. Thanks! Pat
Technical SEO | | PatBausemer0 -
Wrong page version in the index
Hi, my site is currently accessible through URL with and without www. The Version with www has 10 times more Backlinks (PA 45 vs 38) but is not listet into the google Index. As far as I know there was never made a google Webmaster account or declared otherwise the version without www to be 'cannonical'. Basically I think that for SEO reasons it would be much better to declare the with www version to be cannonical and redirect the without www version to it. My questions are: Do you have an idea why the with www version is not indexed?
Technical SEO | | Naturalmente
How long does Google usually take to change the version in the index?
Do I risk my site to be thrown out of the index for some days untill the change is made? Thanks in advance.0 -
Getting a citation page indexed
Howdy mozzers, I have a citation on a .govt domain with 2 links pointing to my site. The page is not indexed by Google, bing or yahoo. URL; http://www.familyservices.govt.nz/directory/viewprovider.htm?id=17077 I have tried getting the paged indexed by building bookmark links to it. I have tweeted the url and gotten a few re-tweets for it. But no luck. The page has got no nofollow meta tag. Other listings have been indexed by google. Could someone please advise on means to help me get the page indexed? A strategy that I have not yet tried is submitting a sitemap that includes the external url as I am not sure if it is possible to include url's not part of my domain. Any advice, help would be greatly appreciated. viva le SEOmoz Thanks
Technical SEO | | ihms1