Robots.txt: excluding URL
-
Hi,
spiders crawl some dynamic urls in my website (example: http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/ + http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/open=true) as different pages, resulting duplicate content of course.
What is syntax for disallow these kind of urls in robots.txt?
Thanks so much
-
You don't want to do this in robots.txt. If you serve pages with these parameters, people will inevitably link to them, and even if they're disallowed in your robots.txt file, Google maybe still index them, according to this: "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web."
This is what the rel=canonical tag is designed for. You should use that to tell Google the page is duplicate content of another page on your site, and that it should refer to that other page. You can read (and watch a video) about that here.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multi Keyword URL Ranking at Number 1
Here is part of a URL that takes the local number 1 spot for "implant dentist glasgow" [website] /implant-dentistry-glasgow-scotland/implant-dentistry-glasgow-scotland.html The first /implant-dentistry-glasgow-scotland/ directory or page is protected and presumably just exists for ranking reasons. I am tempted to copy that URL on a client's implant page to compete for the keyword (I believe I have better content). Given that it works well for the other site, can you think of any reason that would that be a bad idea? Thanks very much.
On-Page Optimization | | neilmac0 -
WordPress image urls - need a WP maven
We were having a conversation re urls that are indexed for images that are stored in various media plugins in WP. My question for anyone who is an uberWP person is: What is your opinion re best media storage plugins and how these URLs affect pages on a site for ranking, etc. I realize this is broad, but it is driven out of my concern that I cannot touch everything. When I see a url like this: http://www.drumbeatmarketing.net/wp-content/themes/drumbeat2/img/DB-LOGO-White.png I know there is no way with all the sites and clients we handle that I can get it perfect but this just bugs me for some reason. Should I just chill since it (seemingly) affects so little....?
On-Page Optimization | | RobertFisher1 -
301 redirects, efficiency and dynamic URLs
Hi, I have 2 301 redirect questions. Question 1: I have am working with a designer on the redesign of a website that currently has over 5,000 indexed pages. The majority of these are dynamic URLs from the Stone Locator database. (see below) http://www.domain.com/storelocator.php?zipcode=91784&page=12 How can I efficiently deal with these pages from an SEO perspective when developing the new site? Is there a way to do a bulk 301 redirect to a store locator page, for instance? Question 2: If a rel=canonical tag has been established on a page (www....), is it necessary to add 301 redirects to all of the other versions on: the home page (domain.com , domain.com/index.html, domain.com/index.html, etc.) all other pages with those same extensions ? Thank you for your help! Erin
On-Page Optimization | | HiddenPeak0 -
Rewrite URL for big website
My website is currently have over 1000 000 links indexed by Google , i'm going to rewrite all of these links, is this crazy ? Can someone give me a checklist to avoid going wrong way ? My website: http://www.webtretho.com/forum/ My current url format: webtretho.com/forum/f[forum_id]/[thread-title-thread_id/ Ex: Old URL: http://www.webtretho.com/forum/f90/chia-se-dau-hieu-mang-thai-1357342-new/ New URL: http://www.webtretho.com/forum/chuan-bi-mang-thai/chia-se-dau-hieu-mang-thai-1357342-new/
On-Page Optimization | | firstjames0 -
"Canonical URL Tag Usage" recommendation in SEOmoz "On-Page Optimization" Tool
Here comes another one related to SEOmoz "On-Page Optimization" Tool. The tool says the following about one of our pages: Canonical URL Tag Usage Explanation: Although the canonical URL tag is generally thought of as a way to solve duplicate content problems, it can be extremely wise to
On-Page Optimization | | gerardoH
use it on every (unique) page of a site to help prevent any query strings, session IDs, scraped versions, licensing deals or future
developments to potentially create a secondary version and pull link juice or other metrics away from the original. We believe
the canonical URL tag is a best practice to help prevent future problems, even if nothing is specifically duplicate/problematic
today. Recommendation: Add a canonical URL tag referencing this URL to the header of the page. Let's say our page is http://www.example.com/brands/abc-brand and on its header we'll place the following tag: Is this correct? I thought the canonical tag was meant for duplicates of the original page, for example: http://www.example.com/brands/print/abc-brand href="http://www.example.com/brands/abc-brand**?SESSID=123** Thanks in advance.0 -
URL STRUCTURE & RE-WRITING
1.) I need a suggestion from you, please help. How should be the Directory/URL structure if I am offering servicves in many cities of UK/US. /<service>/
On-Page Optimization | | younus
/<service>/<state>/
/<service>/<state>/<city>/ OR /<service>/
/<state>/<service>/
/<state>/<city>/<service>/ Thanks for your time.</service></city></state></service></state></service></city></state></service></state></service></service> What exactly mean by 'URL enforce writer' to rewrite the URL.0 -
How long is too long for domain URL length?
I noticed one of the negatively correlated ranking factors was length of URL. I'm building a page from scratch, we are trying to rank for 'Minneapolis Fitness' and 'Minneapolis Massage'. Is www.minnnepolismassageandfitness.com just ridiculously long? Or does the exact match outweigh the penalty for URL length?
On-Page Optimization | | JesseCWalker2 -
How do you see a list of URLs with duplicate page titles?
When looking at the Duplicate Page Title report, the Other URLs column has various numbers that presumably indicate the number of pages that share the same title. When I click on one of these numbers, say a URL that shows 4 in that column, the next page reports "No sample duplicate URLs to report". Why isn't it showing me the other 3 URLs with the same page title?
On-Page Optimization | | jkenyon0