How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help recover lost traffic (70%) from robots.txt error.
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change. Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated. On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL? Thanks,
On-Page Optimization | | akin671 -
Harms of hidden categories on SEO
On our website we have some invisible/hidden categories on our site. Can anyone advise whether these are harmful in terms of SEO?
On-Page Optimization | | CostumeD0 -
Category descriptions and magento
Hi Any eCommerce experts out there, please? I run a magento store and am looking to improve the performance of my category pages in the listings, to aid this I have created category descriptions - a hundred or so words about what, for example, a Maxi Dress is, an example can be seen here http:// yochic.co.uk/dress-267864/ maxi.html (I have broken the link as not sure on the linking policy of the forum). I have noticed, however, that this category description is appearing all on sub pages of the section, so it is on page 1,2,3,4 etc. Is this likely to lead to me receiving a penalty for internal duplicate content? The store uses rel="next" in all the category pages, so shouldn't that help limit any problems caused by the duplication of the page descriptions? I notice that when I search for a sub page, 2, 3, 4 etc in Google using the url of page two http://yochic.co.uk/dress-267864/maxi.html?p=2 it returns the first page, so am i right in thinking with the tags that Google sees all the sub pages as being page one. In which case, doesn't that limit my ability to rank for the products listed on pages 2+? Long story short, the site cannot rank it's product pages because the vast majority of the product descriptions are 100% duplicate because the site is an affiliate fashion store. Many thanks Carl
On-Page Optimization | | carl_daedricdigital0 -
Latent semantic Indexing - Does this help rankings/relevance?
Hi, Does semantically related words to the target term on a page help with rankings/relevance? If your after the term 'PC Screen' and you use the term 'PC Monitor' will go make the connection and also reward you because of the relevance? Anyone do this and have you seen any positives? I've just started to try this out lately and have been combining it with Wordle.net to give me an indication of where the content piece is heading and how aggressive the content leans towards certain words (makes things a little more interesting then calculating densities).
On-Page Optimization | | Bondara0 -
Is this good practise for SEO: www.example.com/testing-once/testing-once.aspx?
My web developer is telling me that it is best practice for SEO to place your keyword in your URL twice. For Example: www.example.com/testing-once/testing-once.aspx. I'm sure that this is not true, could you please help?
On-Page Optimization | | CoGri0 -
Doubt about correct no. of categories on my blog
I have site wide 25 categories on my blog but I dont know whether I should keep few categories only like 6-8 or having 25 categories is ok? Plz help
On-Page Optimization | | ksbnok0 -
Product merchandising category creation
I work on a site selling clothing, and we break up our clothing into categories of types of clothing. Right now in my accessories, I have an other category, which cover the miscellaneous items that there aren't enough to warrant their own categories. I was curious what people thought about further breakdowns of this category. Do I create a category that only has one item in it? Is there a certain threshold of number of items which should signal that these items need their own categories or length of time they're expected to be available to buy? Right now, I'm not targeting the one-off items for SEO purposes because we tend not to carry them from season to season or continue with them long enough to be the best use of my SEO time.
On-Page Optimization | | kennyrowe0 -
Tool for Generating Sitemap/ URL List
HI, I'm looking for a tool that'll generate a URL list for a site. I looked at this thread here http://www.seomoz.org/q/online-sitemap-generator which came up when I searched for sitemap generator. However, I don't need a sitemap per se, and I don't need to submit it to Google - just a list of pages is what I need.If it updated automatically, that would be useful as well. Anyone know of a tool, on or offline? Or anyone used Xenu and know if it's what I'm looking for? Or is there a simple solution that I'm missing? Thanks.
On-Page Optimization | | 5225Marketing0