How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz bar not working on https://www.fitness-china.com/gym-equipment-names-pictures-prices
Moz bar not working on our website about gym equipment names https://www.fitness-china.com/gym-equipment-names-pictures-prices How long fix it?
On-Page Optimization | | ahislop5740 -
Aggregator/comparitor site outranking us
Hi I would like to know if anyone has experience with trying to outrank an aggregator/comparitor website. We are being beat by one that also includes our range of products in their comparisons and I was wondering if there was a smart way around this?
On-Page Optimization | | Discovery_SA2 -
SEO Onpage (where I need to put the H1/2/3)?
Hi everybody, I'm working on my new website, and I have a few questions about the Headers (<h 1="" 2="" 3="">)
On-Page Optimization | | JohnPalmer
please see the attached image, I create a mock-up for you, please tell me which H I have to put on the title.
(** most of the articles will be without sub-titles so...it will be dynamic, but try to take a look on the "sidebar" and "menu link".</h> 6ER0EpK0 -
The effect of having CR LF HT commands in a <title>tag</title>
Hello. I am looking at a customer site with a CMS system that is controlling the population of the meta TITLE Currently it has the TITLE set as this <title>(CR)(LF)<br />(HT)Site Details REMOVED - Customer name REMOVED(CR)(LF)<br /></title> Naturally, we would prefer it to be <title>Site Details REMOVED - Customer name REMOVED</title> What affect would these commands have in the title! Google shows their title when you Google the company website... so I guess it can see it .... but GA "Top Site Content" widget shows it as blank ? Any ideas? Cheers
On-Page Optimization | | BinaryTris0 -
Canonical URL, cornerstone page and categories
If I want to have a cornerstone "page", can I substitute an actual page with a category archive of posts "page" (that contains many posts containing the target key phrase)? This way, if I make blog posts about a certain topic/ key phrase (example "beach weddings") and add a canonical URL of the category archive page to the individual posts, am I right then to assume google will see the archive page as the cornerstone page (and thereby won't see the individual posts with the same key phrase as competing)?
On-Page Optimization | | stephanwb0 -
Summarize your question.Images being seen as duplicate content/pages
My images suddenly are appearing in my crawl reports as duplicate content, without meta tags, this happened over night and cant figure out why.
On-Page Optimization | | RBYoung0 -
Seeking a Google Penalty / Panda & Penguin recovery expert
Hi folks, I've been dealing with an online travel agency who came to me looking for content marketing services. One look at their analytics & GWT was all I needed to see they have some serious site cleanup to do before they can start spending money on our services. Their search traffic floored with the first Panda update back in Feb 2011, and they've gone through all sorts of turbulence with the Penguin updates too. There are no messages in GWT, but I suspect their previous SEO guy might have deleted them to cover his tracks. I did a quick look at their link profile and there's all kinds of junk in there, they have dupe content all over the place and the whole thing needs cleaning up. In other words, it's a mess. I'd like to win some business from them, but first they need to talk to a Panda/Penguin recovery specialist. If you're interested, drop me a private message and I'll put you in touch with them. Thanks, Matt.
On-Page Optimization | | MattBarker0 -
Directory Structure
Hi, We are creating a new content directory for online courses hosted on our site. Like a typical directory, we have high level categories and then more granular subcategories. A course will typically only be in one high level category and then multiple subcategories. What would be the best URL structure for an individual course? Should we force users to pick one 'master' subcategory that gets included in their URL? Or should we just not include the subcategory at all in the URL? Right now we've been thinking about: OurUrl.com/upper-category/sub-category/course-title or OurUrl.com/upper-category/course-title
On-Page Optimization | | mindflash0