Robots.txt: excluding URL
-
Hi,
spiders crawl some dynamic urls in my website (example: http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/ + http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/open=true) as different pages, resulting duplicate content of course.
What is syntax for disallow these kind of urls in robots.txt?
Thanks so much
-
You don't want to do this in robots.txt. If you serve pages with these parameters, people will inevitably link to them, and even if they're disallowed in your robots.txt file, Google maybe still index them, according to this: "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web."
This is what the rel=canonical tag is designed for. You should use that to tell Google the page is duplicate content of another page on your site, and that it should refer to that other page. You can read (and watch a video) about that here.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When using long-tail keywords, should you exactly match for the url or delete "in" "to" etc.?
long-tail keyword - "seizures in adults with no history" Should you include "in and with" in the url?
On-Page Optimization | | Moleculera0 -
Long URL's
So I'm super new at SEO and learning a lot. I'm a small business owner and enjoy doing it myself. Are long URL's good or bad? Like this: http://www.farnorthkennel.com/german-shepherd-puppies-the-girls/long-haired-german-shepherd-puppies-lava Is that too long? The german-shepherd-puppies-the-girls is an actual page with actual content. Do those hurt me?
On-Page Optimization | | Joshlaska0 -
Should ".html" in the URL or not?
Does URL like /wedding-dresses.html/f/price/2,100 with "html" in the middle do harm to SEO? Should I drop the ".html"?
On-Page Optimization | | StevenRoland0 -
Disallow a spammed sub-page from robots.txt
Hi, I have a sub-page on my website with a lot of spam links pointing on it. I was wondering if Google will ignore that spam links on my site if i go and hide this page using the robots.txt Does that will get me out of Google's randar on that page or its useless?
On-Page Optimization | | Lakiscy0 -
Using magentos own url re-writes
We are changing ecommerce platforms. Is it best to use magentos own url re-writes to redirect every page of a site from its old url to it new one?
On-Page Optimization | | LadyApollo0 -
Does having a "+" in a URL hurt SEO? Would much value be gained changing it to a hyphen?
There's a site that contains "+" signs in the URL in order to call different information for the content on the page. Would it be better to change those to hyphens (-), or not that much value will be gained, so leave them as is? Thanks!
On-Page Optimization | | MitchellStoker0 -
How to Define Best URL Structure for Product Pages?
I am working on my website to edit structure with help of Google's search engine optimization starter guide. There is really good instruction to define URL structure which help us to perform well over Google's organic search. I have resolved issues regarding category pages but, I have confusion to define best URL structure for product pages. My website's product page URL structure is as follow. http://www.vistastores.com/marketumbrellas-californiaumbrella-slpt758-f13-red.html http://www.vistastores.com/homefurniture-winsomewood-93630.html URL structure is constructed with following terms. 1. Root Category Name (Market Umbrellas or Home Furniture or ....) 2. Brand Name 3. Manufacturer Part Number I am not happy with this structure and also not performing well over Google's organic search. I am thinking to include product name or title tag in URL after root domain. But, it may create very long URL and create issues in organic search display. Does it really matter to perform well over Google's organic search? How can I define best URL structure for product pages?
On-Page Optimization | | CommercePundit0 -
301 redirect OK for a newer version of a page that is a different url?
I have about 500 products with multiple urls for the same product, but different versions. I sell wine and have a different page for each vintage. I've decided that is not the best way to go, and want to point the older vintage pages to the latest version page, and make that the only page for the product as time goes on. Do I have to put a link in the text from each older page to the newer, or can I use a 301 to redirect them to the new page? I don't want google to think I'm pulling something funny.
On-Page Optimization | | JeanYates0