Can I safely block my product listing from search? Does it even make sense?
-
Hi,
I've an ecommerce website with more than 50k urls and only 10% or so are getting crawled regularly by Google.
Product listing pages represent roughly 80% of these 50k pages.Trying to improve this, I was thinking to remove altogether all (most?) of my product listing from search (via Robot.txt) to keep only the product pages themselves and the product categories.
My organic situation since Jan 2019:
Users: 2,300,000 (of which 9% are visiting product listing pages)
Page views: 8,000,000 (of which 5% are product listing pages).Am I about to unleash armageddon (or more like harakiri) on my website by doing so or actually get Google to crawl much more relevant resources (product pages, product categories, blog content and so on)?
Thanks,
G -
Have had a lot of success with that kind of deeper logic in the past, you can usually quite easily create such rules using robots.txt wildcards
-
Thanks for your answer and the link, that's actually very useful.
What I mainly struggle with is to understand what I can prune/not and based on what criterias. And the article you've linked is helping a fair bit on that aspect.I'm thinking to start on multiple filter pages with a rule such as "block product listing pages from being indexed if at least 2 filters have been selected". And then see how it impacts the site and my crawl budget.
-
Hello GhillC,
I think we need to agree on terminology first, but it sounds like you can safely limit some of Google's access. Some people call your "product listing pages" either "refinements", "facets", or "filters". When I read "product listing pages" I typically think of what is also called a "category" page, which is a page listing multiple products. A single product page is often referred to as a product detail page (PDP).
Now that we're on the same page (pun intended), let me know if this article answers your question. It is very dated (2011) but gets the point across, which is that you need to be strategic about which facets/refinements/filters you allow to be crawled and/or indexed: https://mza.seotoolninja.com/blog/building-faceted-navigation-that-doesnt-suck .
One more thing: 5% - 9% of traffic going directly from organic search into a page type would be considered significant for most businesses. When I look at pruning out page types, they're typically responsible for less than 0.5% of traffic from organic search.
-
A product page is a page speaking about a selected product.
Example: a page about the smartphone Samguns Supernova 30A product category is a page speaking about the category, or the type of product it falls into.
Example: smartphone or, if it is too broad, the Supernova lineA product listing is a page listing all the product refined with available criterias such as a product line.
Example: a page listing all the Samguns Supernova phones
Or a page listing all the Samguns Supernova phones with more than 128GB of HD.
Etc.Does it make more sense?
-
Can you explain the difference between the 'products listings' and the 'actual products themselves'?
You say you still want products and product categories to rank, but not product listings. But to most readers, a product listing is usually a product category or product page (so the info seems to contradict itself, which actually it may not do - just needs more explaining)
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which product URL to include in Sitemaps?
Hi Does the product URL's in Sitemaps affect the sub-categories authority too? For example, if I have a product with 2 URL's and which have a canonical tag: **/brands/michael-kors/bags/**jet-set-double-zip-wallet/ **/women/accessories/wallets/**jet-set-double-zip-wallet/ If I make the main URL "/women/accessories/wallets/jet-set-double-zip-wallet/" and set that as the Canonical URL & list that URL in the XML Sitemap, will it also mean the "/women/accessories/wallets/" category will get more authority and increase it's power to rank? Thanks Frankie
Technical SEO | | Frankie-BTDublin0 -
Skip indexing the search pages
Hi, I want all such search pages skipped from indexing www.somesite.com/search/node/ So i have this in robots.txt (Disallow: /search/) Now any posts that start with search are being blocked and in Google i see this message A description for this result is not available because of this site's robots.txt – learn more. How can i handle this and also how can i find all URL's that Google is blocking from showing Thanks
Technical SEO | | mtthompsons0 -
PhantomJS to Make AJAX Pages Crawlable
Anyone have any experience using PhantomJS to return HTML snapshots of AJAX rendered pages? More specifically, does anyone know if Google takes issue with this technique in any way? Interested in learning about this technique? Using PhantomJS to allow Googlebot to crawl your AJAX pages.
Technical SEO | | RyanOD0 -
Removing a URL from Search Results
I recently renamed a small photography company, and so I transferred the content to the new website, put a 301-redirect on the old website URL, and turned off hosting for that website. But when I search for certain terms that the old URL used to rank highly for (branded terms) the old URL still shows up. The old URL is "www.willmarlowphotography.com" and when you type in "Will Marlow" it often appears in 8th and 9th place on a SERP. So, I have two questions: First, since the URL no longer has a hosting account associated with it, shouldn't it just disappear from SERPs? Second, is there anything else I should have done to make the transition smoother to the new URL? Thanks for any insights you can share.
Technical SEO | | williammarlow0 -
Getting listed in DMOZ after a red mark against me?
Hi everyone, Before I start, I want to point out that I realise DMOZ is not everything. If I can't get a link, then I'll accept it, but I'd like to try damn hard to get a link in there if possible as I honestly believe we are a reputable company that deserves to be in there with other companies in our region. So, lets go back to the start: A few years ago, when I was young and foolish, I posted a request on rentacoder (now vworker) asking for someone to submit us to DMOZ for $xxx. If I remember correctly, we accepted someone for the job and they submitted our site. Since then, we've never been accepted. I tried submitting the site a couple of years ago and realised it still hadn't been submitted around a year ago. I thought this was a bit strange, so I did a bit of digging. I ended up talking to someone who was a DMOZ editor. They looked at my listing on DMOZ and basically said that I'd be rejected for 'bribing an editor' - something that I still maintain I didn't do. Never at any point have I said to someone that I would pay them to guarantee us a listing. At the time, I thought there was a 'certain way' to write a submission, but now that I've grown up a bit, I've realised it's fairly easy to get a listing if you write a well written description and are honest - I've done this for many clients sites which are now accepted. Unfortunately, this does mean that It looks like I won't be able to get our agency on DMOZ, but I just wanted to give one last push and see if anyone on SEOMoz knows of any one I could speak to in order to lift this ban? I realise now that it was probably a stupid thing to do, and in hindsight, I should have just waited like everyone else. I hold my hands up to that and I've worked over the last few years to grow our agency into something special for our clients. As I say, I realise dmoz isn't everything, but it is a directory that still holds value and I would like to be linked off there if possible. Any tips on how I may be able to beg for my ban to be lifted would be much appreciated. Kind regards, Lewis PS: I applied to become an editor at one point, this was when I realised something wasn't right - http://www.resource-zone.com/forum/index.php?showtopic=52658
Technical SEO | | PinpointDesigns0 -
Quantity of products
I have an e-commerce site that sells around 1,500 SKU's.. but there are some huge categories that we really never sell anything in. I'm thinking of deleting a lot of underselling products. When I put it like that of course, you'll probably agree - but I'm hesitant because I have to wonder what Google will think of my site going from (say) 1,000 products and unique URL's to 500.. or lower. Removing that many URL's, products and product categories worries me that we may be viewed as a smaller store and have a negative impact. What effect would you expect from removing a lot of products? Who has done this?
Technical SEO | | TellThemEverything0 -
Duplicate content q - Can search engines tell where the original text was copied from?
I was under the impression that when a search engine comes across duplicate content it won't be able to determine which one is the original. Is this not the case?
Technical SEO | | Sparkstone0 -
Best URL Structure for Product Pages?
I am happy with my URLs and my ecommerce site ranks well over all, but I have a question about product URL's. Specifically when the products have multiple attributes such as "color". I use a header URL in order to present the 'style' of products, www.americanmusical.com/Item--i-GIB-LPCCT-LIST and I allow each 'color' to have it's own URL so people can send or bookmark a specific item. www.americanmusical.com/Item--i-GIB-LPCCT-ANCH1 www.americanmusical.com/Item--i-GIB-LPCCT-WRCH1 I use a rel canonical to show that the header URL is the URL search engines should be indexing and to avoid duplicate content issues from having the exact same info, MP3's, PDF's, Video's accessories, etc on each specific item URL. I also have a 'noindex no follow' on the specific item URL. These header URLs rank well, but when using tools like SEOMoz, which I love, my header pages fail for using rel canonical and 'noindex no follow' I've considered only having the header URL, but I like the idea of shoppers being able to get to the specific product URL. Do I need the no index no follow? Do I even need the rel canonical? Any suggestions?
Technical SEO | | dianeb1520