Robots.txt: excluding URL
-
Hi,
spiders crawl some dynamic urls in my website (example: http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/ + http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/open=true) as different pages, resulting duplicate content of course.
What is syntax for disallow these kind of urls in robots.txt?
Thanks so much
-
You don't want to do this in robots.txt. If you serve pages with these parameters, people will inevitably link to them, and even if they're disallowed in your robots.txt file, Google maybe still index them, according to this: "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web."
This is what the rel=canonical tag is designed for. You should use that to tell Google the page is duplicate content of another page on your site, and that it should refer to that other page. You can read (and watch a video) about that here.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does changing/shortening a url hurt SEO?
Hi all, I am in the process of making small optimization changes to my site. I noticed Moz identified quite a few URLs that could be shortened. I intend to shorten these URLs and create 301 redirects to ensure website users land on the right page. My question is, will this change in URL damage rankings and engagement(assuming the URL remains content relevant)? I have read in some places that when creating URL redirects for a change in domain, people saw a dip in rankings and engagement. I, however, am not intending to change the main domain of the site, but rather the URL slug. Any thoughts?
On-Page Optimization | | annegretwidmer0 -
Newbie SEO ?: Does my About page URL have to contain the word About?
New to WordPress and SEO. Built and launched my website last week. The URL was originally domain/about. However, I installed Yoast plugin and it told me "about" was a stop word. So, without too much thought (my first problem), I changed the url (before Google crawled me) to clearwingcommunications.com/storytelling. Since then, I've noticed that sites I know are optimized have their URL with the word "about." So, is this considered a bad practice? My site HAS been crawled at this point. If I change it back to About and do a 301 redirect, does that hurt reporting? Thanks for your help! Christy
On-Page Optimization | | christyr0 -
Googlebot found an extremely high number of URLs on your site:
Website: www.gobol.in Although I have no indexed my search pages by adding /catalogsearch in robots.txt, still we are getting same error again and again Here's a list of sample URLs with potential problems. http://www.gobol.in/catalogsearch/result/index/?category=&mobile_feature=4575_4578&q=panasonic+NR-BU303LH1H+REFRIGERATOR+296+L+GREY&special_price=32%2C456&x=0&y=0 http://www.gobol.in/mobile-and-accessories/mobiles-and-brands.html?manufacturer=4753_3355_455_4435_4720_3407_2412_4728_4784_4790_2010_4789_4376_2469&operating_system_mobile=4612 Please help
On-Page Optimization | | Obbserv0 -
Properly changing title, URL and content for new keywords without harming other rankings.
Hello - We are looking to try to bring up some keywords in the SERPs that we are currently ranking fairly low for. We sell Christening clothing for children and people will use both Christening and Baptism to search for the same thing. We currently rank very high for Christening (#1 on Google for certain combinations) but we are fairly low on Baptism.
On-Page Optimization | | BabyBeauBelle
I am trying to figure out the best way to start getting Baptism up by changing some title, URL and content pages to include more Baptism keywords. My concern is messing with the existing because we rank so well for Christening. Since we are ecommerce we can vary this quite a bit on our products, but again I'm nervous to do so fearing changing the wrong things, too many products etc and in the process of trying to raise one set of keywords (baptism) we harm the other set (christening).
Any advice would be appreciated!0 -
Does Google follow link path or url path when it comes to passing link juice
I noticed something with one of my sites and now I am thinking I made a boo boo (I think) here is what I have On my homepage I have 5 links Link1
On-Page Optimization | | cbielich
Link2
Link3
Link4
Link5 Links 1 - 4 go to a page and stops there. So my URL structure is www.mydomain.com/Link1
www.mydomain.com/Link2
www.mydomain.com/Link3
www.mydomain.com/Link4 So naturally my link juice passes down to these links evenly. Link5 also goes to another page, but on that page I have more links that go down further. www.mydomain.com/Link5 -> more links On page Link5 I have links that go to more pages, BUT my URL structure for these pages go like this Lets say on Link5 page I have another link that goes to AnotherLink1, AnotherLink2 and AnotherLink3 When you click on those links it takes you to those pages just fine, BUT my URL structure is like this www.mydomain.com/AnotherLink1
www.mydomain.com/AnotherLink2
www.mydomain.com/AnotherLink3 Basically I put all the "AnotherLink1-3" in the root directory as well. My question is concerning how Google passes the link Juice from my pages and if it is passing based on the path of the links and how they point to those pages, or do they pass link juice based on the URL structure. So since "AnotherLink1-3" is located in the root directory am I dividing my link juice from my home page to all the links as well based on the URL structure. For instance www.mydomain.com/Link1
www.mydomain.com/Link2
www.mydomain.com/Link3
www.mydomain.com/Link4
www.mydomain.com/Link5
www.mydomain.com/AnotherLink1
www.mydomain.com/AnotherLink2
www.mydomain.com/AnotherLink3 Do I need to change my path for Link5 page to www.mydomain.com/Link5/AnotherLink1
www.mydomain.com/Link5/AnotherLink2
www.mydomain.com/Link5/AnotherLink3 ?0 -
Can bad text URLs hurt pages?
If you have some pages that contain plain text URLs (not anchored links) that used to be good URLs, but are now bad, either because the website shut down or because it has been acquired by someone else and is now parked (or worse) - are those URLs enough to cause quality problems? For example: This information was brought to you by Waymaker http://www.waymaker.net These aren't the only ones. And yes, I know I should fix them, but there are probably 10,000 pages like it. I will fix them, but its not something I can do in a few minutes. (this one is easy to fix programmatically, but others are a lot more complex) So my question is: do you have actual experience that these are bad enough to cause ranking problems (making them low quality)
On-Page Optimization | | loopyal0 -
3 Different Home Page URL's Being Indexed?
Hello Everyone! I own a dog supplies eCom site on the x-cart platform. I recently upgraded to 4.4 version about 3 weeks ago and am noticing 3 different home page URL's getting indexed and ranked: /
On-Page Optimization | | k9byron
/home.php
/home.php?cat= I dont know why this is happening and I dont claim to be an expert SEO but know this cant be good! I am seeing high rankings on certain terms for all 3 URL's. Has anyone seen this before and can anyone give me any feedback on this and how it may be effecting my sites ranking in the future? Thanks in advance!
Byron-0 -
Can someone please help me identify where all these URLS to my homepage are coming from?
Hi. I installed the SEOmoz toolbar for Firefox, and analyzed my home page, then clicked on 'get a full site analysis at Site Explorer'. This is what came up: http://www.opensiteexplorer.org/www.frs-solutions.com%252Fcontent%252Fhome/a!links?src=mb I hope that link works. If not, the URL is www.frs-solutions.com Anyway, there are about 57 different URLS within my site all pointing to my homepage! I have no idea where they are coming from. Can someone with an experienced eye take a quick look and tell me what I might be up against? Thank you!
On-Page Optimization | | aprilm-1890400