How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best meta description for Category Pages, Tag Pages and Main Article?
Hi, I want to index all my categories and tags. But I fear about duplicating the meta description. for example: I have a tag name "Learn Stock Market", a category name "Learning", and a main article "What is Stock Market". What is your suggestion for meta description of these three pages that looks great for seo google?
On-Page Optimization | | mbmozmb0 -
Does a / at the end of a URL create a duplicate page?
Hello, I have just used (the amazing) Screaming Frog to check my site and it is listing the two following pages as having duplicate titles, making me think it is seeing them as duplicate pages. http://zenplugs.com/zenplugs-molded-earphones/ http://zenplugs.com/zenplugs-molded-earphones Do I need to redirect one of these? Thanks in advance! Toby
On-Page Optimization | | T0BY0 -
My report is showing duplicate titles.. for http://www.mysite.com/ & http://mysite.com
Is this because I setup my campaign wrong in seo moz or should I have my site setup differently to forward all traffic without the www to the www version? Or vise versa? Thanks for any advice!
On-Page Optimization | | Fridaythe15th0 -
Product category content!? what should it include?
Hello everyone!, I consider myself a rookie... so... please, excuse me if this is super basic or dumb!. I'm working on a ecommerce web (family business!)... and i've got this doubt. Say you've got architected your site this way...: site.com/category
On-Page Optimization | | jleandroperez
site.com/category/model_1
site.com/category/model_2 I'm mainly interested in getting the category webpages to rank high. The problem i've got is... what to put in the CATEGORY webpage!. Suppose you sale office furniture... and the category is 'chairs'... if you add content there, it won't be useful. What do you suggest me to add there?. ====== NOTE: My 'categories' webpage is split vertically, so you've got an image gallery on the left, and the product description on the right. So all of my product pages look a bit alike... and the 'category' itself has a placeholder on the right. I suspect that's why i'm not getting good rankings! THANKS in advance.0 -
Using fathead page keywords for directories and as a red herring to competitors
I'm fairly new to SEO and I have been reading a lot on here and the SEOmoz guides over the last few days, finding it very interesting. I am wondering about page keywords, I read that the engines no longer use them. In this thread they say they still use them because of directories. http://www.seomoz.org/ugc/why-i-still-use-meta-keywords#new-comment So I was wondering, because directories are always smaller scale than search engines would keywords that are normally high competition have more clout in them? If so then using them could be misleading to competition if they think they are the actual keywords? or might this contradict between your actual chosen keywords in the directories backlink anchor text or something?
On-Page Optimization | | Zoolander0 -
How do I do a 301 Redirect in IIS 7 from http://www.freightmonster.com/index.html to http://freightmonster.com/index.html when I don't have a physical page to redirect?
I'm trying to get rid of my Rel Canonical links and use the 301 Redirect instead.
On-Page Optimization | | FreightBoy0 -
Installing a site on top level domain directory VS deeper directory
How important it is for seo to install a site on a top level directory vs deeper directory? example: www.mysite.com VS www.mysite.com/catalog many eCommerce scripts such as oscommerce and cre loaded will install by default to /catalog and that's what I've been doing for most of my customer. does it mean it will be harder for them to get good seo results? Thanks in advance for the input...
On-Page Optimization | | zigi0 -
Top Level Category Pages
On certain sections e.g. Top Level Brand Pages i would like to include history, images of products in use, videos, and possibly even a timeline of the manufacturers history. Being that its a store this would push the products quite a bit lower on the page. Can i still have a great positive effect if its below the product listings?
On-Page Optimization | | Anest0