How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are there detrimental effects of having multiple robot tags
Hi All, I came across some pages on our site that have multiple robot tags, but they have the same directives. Two are identical while one is for Google only. I know there aren't any real benefits from having it set up this way, but are there any detrimental effects such as slowing down the bots crawling these pages? name="googlebot" content="index, follow, noodp"/> Thanks!
On-Page Optimization | | STP_SEO0 -
Form Only Pages Considered No Content/Duplicate Pages
We have a lot of WordPress sites with pages that contain only a form. The header, sidebar and footer content is the same as what's one other pages throughout the site. Each form page has a unique page title, meta description, form title and questions but the form title, description and questions add up to probably less than 100 words. Are these form pages negatively affecting the rankings of our landing pages or being viewed as duplicate or no content pages?
On-Page Optimization | | projectassistant0 -
Would Changing the Titles of Root Categories Be Bad?
I have researched some more effective keywords to change my root category titles to. I am wondering if it would be a bad idea to change these titles considering all the things that could go wrong. From what i'm gathering there are a LOT of things that can go wrong but at the same time these things do need to be changed sometime! Is this a good or a bad idea & why? What could go wrong? Should I try changing the category titles one at a time instead of risking every one of my keywords / category titles not working out in the serps right away?
On-Page Optimization | | Mike.Bean0 -
Will google see you bold/emphasis words if done in css?
We do not use header tags in our website. I understand bolding or emphsising words can be equally as effective but if done so in css will the google crawlers and spiders etc be able to put a weighted value on this style of code???
On-Page Optimization | | gsbureau0 -
Keep our category navigation in tree structure but move our URLs to a more flat structure. Good plan?
In our Magento store, products are arranged into categories, subcategories and so on. We typically have 3 or 4 layers of category depth. This makes it nice and easy for customers to find stuff, but it means that the end categories have massive long urls. I'd like to keep our category tree structure in place from a navigation point of view, but I feel the url structure is pushing some important stuff to the back of the shop as it were. We have something like 200 categories in total. So, assuming every individual category has an a unique name, I'd like to rewrite the urls so that: ourshop.com/car-parts/
On-Page Optimization | | DWJames
stays as
ourshop.com/car-parts/ ourshop.com/car-parts/suspension/
becomes
ourshop.com/suspension/ ourshop.com/car-parts/suspension/springs
becomes
ourshop.com/springs/ ourshop.com/car-parts/suspension/springs/thismake-lowering-springs
becomes
ourshop.com/thismake-lowering-springs/ and so on....
I'll need some custom magento URL rewrite work done, but that's another story. The real question is whether you guys feel this is worthwhile? Are there any other stores with a deep categorised navigation structure, but a flat url structure? thanks, James0 -
Re SEO, Is it better to have a mega menu or one split into categories?
Hi From an SEO perspective is it better to have 1 large listed menu - over 100 internal links - or say a much smaller menu split into say 10 categories - so 10 links.
On-Page Optimization | | joeprice
This menu would be appearing on nearly all pages but not the home page In terms of the user experience, in my case, I have a feeling having 1 large menu is possibly preferable as it reduces the number of clicks the user makes to get to the desired information and categorisation can be a bit subjective, but i'd like some advice from an SEO perspective. Thanks0 -
Related keywords in title/H1 tag
Hi, I am trying to improve our rankings for pages with photos/images. For the title is it benificial to include keywords that are almost identical in nature? For example: "Brad Pitt Photos and Images" In Google trends photos and images are both commonly used words so including both seems like it would help. When I search for each one separately in Google (Brad Pitt Photos vs Brad Pitt Images) different sites are returned (except for the ones that include both image and photos keywords). I had read that Google knows that Images and Photos mean the same thing, but the search results do vary. I know stuffing all related combinations isn't good, but selective phrases seem to make a difference. Just want to verify if this makes sense. Thanks
On-Page Optimization | | NicB10