Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it OK to 301 redirect 1000s of duplicate random URLs to homepag?
Hello, We found a critical error in our site internal link structure and the way Google indexes it. Website has 1000s of URLs that are basically 50% match to homepage. They all start the same example.com/category/random/random I can do a redirect match and 301 them to homepage. This way 1000s of bogus url are not indexed and no value given. Is it OK to redirect so many URLs to homepage? Platform is creating these URLs because of search query, where it adds all site content to one page. Currently this search page /category / has own canonical and all those duplicate content URLs have canonical to that /category /. To fix my plan is to a. Remove canonical from /category / that way all those duplicate URLs don't have it either. B. Redirect match all URLs that have /category / in them to homepage. (this is most important page where 50% of that content is and should be the main page). Is this plan ok?
On-Page Optimization | | advertisingcloud1 -
Specific Ranking Mystery
Hello Mozzers I don't understand this specific ranking example, maybe somebody can help? For the serach term "elipslife" on Google Switzerland a client's page ranks around position 13. A competitor's page ranks at 9. And I don't get why: the competitor's page has no content and its DA is lower. Its PA is higher so I set a link in the client's homepage navigation a couple of weeks ago to lift the client's page PA above that of the competitor. This is not yet reflected in OSE currently however. Any ideas as to what may be the cause? Thx, Chris
On-Page Optimization | | ChrisCronimund0 -
How to transfer old WP blog to new URL
I have a 9 year old WP website with a WP blog which is still getting 300+ new visitors a day even though I have not written a blog for 5 years and have not updated content. Some posts have over 25,000 links. However the Moz analytics is fraught with significant errors-404 redirects, page not found, dup content, no metatags, title too long etc. I was totally inexperienced 5 years ago and made many errors. However the basic content was sound and still is producing new visitors. I am starting a new ecommerce website using the same name but the URL and server will be different. I want to transfer my WP blog to the new site. I am concerned however that bringing the posts over can create the same errors on the new site. If I update all of the blogs on the old site using Yoast before transferring the blog to the new site will that help. I suppose I could check those flagged dup content and only transfer one of that category?
On-Page Optimization | | wianno1680 -
Content with changing URL and duplicate content
Hi everyone, I have a question regarding content (user reviews), that are changing URL all the time. We get a lot of reviews from users that have been dining at our partner restaurants, which get posted on our site under (new) “reviews”. My worry however is that the URL for these reviews is changing all the time. The reason for this is that they start on page 1, and then get pushed down to page 2, and so on when new reviews come in. http://www.r2n.dk/restaurant-anmeldelser I’m guessing that this could cause for serious indexing problems? I can see in google that some reviews are indexed multiple times with different URLs, and some are not indexed at all. We further more have the specific reviews under each restaurant profile. I’m not sure if this could be considered duplicate content? Maybe we should tell google not to index the “new reviews section” by using robots.txt. We don’t get much traffic on these URLs anyways, and all reviews are still under each restaurant-profile. Or maybe the canonical tag can be used? I look forward to your input. Cheers, Christian
On-Page Optimization | | Christian_T2 -
Replacing website, same URL, lose ranking
We had to do a rush job for a client and get a website up very quickly. We had to just replace the existing files on the server, but URL stayed the same. Now "teeth whitening glasgow" which they were number #1 for isn't coming up. Know there is still some on page work to be done - but the link profile should still remain the same? Just looking for some advice to get back to where they were - should we do a website change of address or just verify the site in Webmaster and get all the on page done? Thanks, Laura
On-Page Optimization | | lauratagdigital0 -
URL Rewrite
(By Google Traductor) Hello, I wanted to ask about some changes that we are evaluating for the issue of passing the url with variables to be more descriptive, for example: http://www.agroads.com.ar/detalle.asp?clasi=139592 tohttp://www.agroads.com.ar/humedimetro-para-cereales-draminski-gmm-139592.html In this case corresponds to the breakdown of a product if you have long published andcan be well positioned to change the title of this position would be lost unless youmanage it with a 301, as one would manage when you have more than 30000 products and title may change several times? There are tools to manage this? Finally, we must apply this to all listed with their respective filters, recommends doingtheir part with 301 redirects and analyze what funciene well to continue with the rest or implement a complete change? I hope I can bring a little light to implement this. Greetings and thanks! Roberto
On-Page Optimization | | romaro0 -
Seeking URL Advice
Hey Moz Community, I'm looking for some URL structure advice for a new directory of a website. We're trying to rank for the term 'internships abroad in <country>'</country> We have roughly 100 pages targeting specific countries. Right now the URL structure is www.gooverseas.com/internships-abroad/china, but some of my colleagues believe this structure would be better: www.gooverseas.com/internships-abroad/intern-in-china. I personally prefer the shorter structure, but we couldn't come to any agreement so we thought we'd pose the question to the community. Any thoughts? Thanks!
On-Page Optimization | | dunklea0 -
Tool for Generating Sitemap/ URL List
HI, I'm looking for a tool that'll generate a URL list for a site. I looked at this thread here http://www.seomoz.org/q/online-sitemap-generator which came up when I searched for sitemap generator. However, I don't need a sitemap per se, and I don't need to submit it to Google - just a list of pages is what I need.If it updated automatically, that would be useful as well. Anyone know of a tool, on or offline? Or anyone used Xenu and know if it's what I'm looking for? Or is there a simple solution that I'm missing? Thanks.
On-Page Optimization | | 5225Marketing0