Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirect chains error on the home page URL
Hello Everyone, I'm getting redirect chains error on the home page URL:
On-Page Optimization | | Nikhil_Falcon
http://ebitdacatalyst.com in Wordpress. I've checked my redirection list in the plugin, and haven't found any redirections on http://ebitdacatalyst.com. Can anyone please help me in solving this issue? I don't know where from it's coming.0 -
301 and the base URL
Hi, please bear with me as I'm pretty new to all this! I've my base URL but obviously want to add keywords to it for seo purposes. Should I redirect from the base URL to the URL with the keywords appended? So my landing page goes from say www.moz.com to www.moz.com/keywords-here. If I do that, should I replicate all the meta data (descriptions etc) on the original landing page? Or does it not matter? Thanks, Nick
On-Page Optimization | | nickwoodward0 -
Product URL
Hey Mozzers, Nice quick and simple one for you. Which of these 2 options is better for SEO and userbility and why domain.co.uk/productname.html
On-Page Optimization | | ATP
domain.co.uk/shop/category/product.html The top one stops stops any funky problems with magento making 2 paths to the same product but the second option feels more natural and helpful to the user. I feel both a valid but I would like some opinions please0 -
Long URL's
So I'm super new at SEO and learning a lot. I'm a small business owner and enjoy doing it myself. Are long URL's good or bad? Like this: http://www.farnorthkennel.com/german-shepherd-puppies-the-girls/long-haired-german-shepherd-puppies-lava Is that too long? The german-shepherd-puppies-the-girls is an actual page with actual content. Do those hurt me?
On-Page Optimization | | Joshlaska0 -
Should I use www in my url when running On-Page Report Card?
When creating a On-Page Report Card I get 2 different results when using a WWW and without for my url. What is best?
On-Page Optimization | | thomas.wittine0 -
One Page Website vs. Multipage Site, if you want to target one specific Keyword only.
Hello! suppose I want to start a website about, let's say spray adhesives. My aim is to rank on the first page for the keyword "spray adhesive". I don't care about my ranking on more specific keywords like "Tesa spray adhesive" or "3M spray adhesive". My ranking for more general keywords like "glue" is unimportant, too. So I thought about creating a single-page website, that writes about spray adhesives, the pros & cons of every manufacturer, and shows the best discounts for spray adhesives. Each section can be accessed through a top-navigation, that links via anchors to the individual sections. The page will be updated every day On the other hand, i could create a blog and write an article for every specific spray adhesive. So I would have a home page that lists the latest articles for every product, with titles like "3M spray adhesive CreativeMount", "3M spray adhesive SprayMount", "Tesa Spray adhesive" ... I will write one article every day What do you think would be the better strategy? Is there a risk to create competing articles for the keyword "spray adhesive" and thus rank lower if I go with the blog strategy? On the other hand, does google rate singe-page websites lower, because google thinks those websites are less valuable than websites with many pages for the same topic? Thank you ver much for you help in advance!
On-Page Optimization | | MGMT0 -
404 crawl errors with all url+domain
We have 187 crawl 404 errors. All urls on web make a 404 error that this http://www.domain.com/[.....]l/www.domain.com all errors added to the url, the url domain I put an example gestoriabarcelona.com/www.gestoriabarcelona.com
On-Page Optimization | | promonet
gestoriabarcelona.com/tarifas/www.gestoriabarcelona.com
gestoriabarcelona.com/category/noticias/page/7/www.gestoriabarcelona.com
gestoriabarcelona.com/2012/08/amortizacion-de-unaconstruccion/
www.gestoriabarcelona.com
[..] I don't know where can i find to solve errors Anyone can help me? Thanks0 -
"On Page" report says 2 rel canonical urls-how do I fix that?
I am reviewing my On Page scores and I'm not getting a perfect score bk of this notice: No More Than One Canonical URL Tag Moderate fix <dl> <dt>Number of Canonical tags</dt> <dd>2</dd> <dt>Explanation</dt> <dd>The canonical URL tag is meant to be employed only a single time on an individual URL (much like the title element or meta description). To ensure the search engines properly parse the canonical source, employ only a single version of this tag.</dd> <dt>Recommendation</dt> <dd>Remove all but a single canonical URL tag</dd> <dd>HOW do I fix that?</dd> <dd>I am using Platinum seo plugin which I have checked "Use canonical urls" and the page in question is</dd> <dd>http://adderalldosage.net/general-adderall-dosage/</dd> </dl>
On-Page Optimization | | ccare7230