Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disadvantages of Migrating Website to New URL
Hi There, I am currently struggling with the ranking of my website. No matter how many initiatives I try (backlinking, blog commenting, social posting, etc.) I can't seem to make any progression in Google Search. I've done competitive metrics through Open Site Explorer and can't seem to really find the reason why my site is not ranking as well as my competitors. The only one possible glaring element I've thought about is my website URL. This company is in the heating and cooling industry and majority of my competitors have either "heating" or "cooling" or both in their website URL's but mine does not. Does anyone have any thoughts or recommendations on if changing my URL and then redirecting my current URL would be a step in the right direction help me to climb the rankings in Google Search? Thanks!
On-Page Optimization | | MainstreamMktg0 -
If Product Pages Perform Well In Google then Is It Possible That Category Page Can perform Well In Google?
Hi All, For my ecommerce site I have optimized my product pages very nicely like good images, detailed information about products, good reviews, implemented schema for my product and reviews and very perfect onpage. Now my query is if my products pages performing well in google then there are chances that my category page rank well in google too? Thanks!
On-Page Optimization | | wright3350 -
Canonical URL Category and Tags
Hello, I would like to know that I want to use both category and tags in my blog StylishMahi. If I index both category and tags, should I use canonical URL tag to pass referring to main category. As I want more my categories in SERP results ranking higher? I have also attached a picture. Can someone please confirm? Photo by Moz ZigdWMx
On-Page Optimization | | PratapSingh0 -
URL / Meta info for Author bio pages
When you create a biography page for authors that point back to their G+ account, do you include any other signals in the URL other than the authors name? For example, would you use www.domain.com/sam-spade or www.domain.com/sam-spade-biography, etc? Similar question for the meta description. Do you add any signals there? Or, is rel=author strong enough. Best,
On-Page Optimization | | ChristopherGlaeser
Christopher0 -
URL parameters
Hello, Currently, I paginated a content to 5 pages eg: http://abc.com/faqs.html?&page=2 Is it right? and how to check it is correct or not?
On-Page Optimization | | JohnHuynh0 -
Keyword repeats/presence in url's & over-optimisation
Hi I'm about to launch a redesigned site and worried about overdoing kw presence on-page, primarily using in url's since will already be using kw in titles as well as page content. What's current thinking re over optimisation: If kw is in titles and page content is it best not to repeat again in url structure i.e. less is more, even though this will cause things like SeoMoz on-page grade score to fall, or better to keep them/add them ? Personally i think it makes sense to include kw in url again since helps make the page relevant, and so long as matches the content should help as opposed to hinder rankings for the pages target keyword. However when i look into this some say don't do this since is over-optimisation The sites generally ranking quite well for its target kw which i obviously don't want to lose after re-launch & hopefully improve further, in the case of this example they are 'Sports Centre Services' & 'Sports Centre Equipment Rental'). The sites current url structure is similar to this below example: frankssportscentres.com/services/sports-centre-equipment-rental Would it be better to keep following existing/above format or to go with either of the below options i.e. more kw rich urls or less: frankssportscentres.com/sports-centre-services/sports-centre-equipment-rental Or frankssportscentres.com/sports-centre-services/equipment-rental Or even less frankssportscentres.com/services/equipment-rental Many Thanks in advance for any helpful comments Cheers Dan
On-Page Optimization | | Dan-Lawrence0 -
Creating Authority and choosing URL's
Creating Domain Authority and choosing URL's: A: What is better if you want to get higher Domain Authority? Choose keyword.domain.com or www.domain.com/keyword when other sites link to it? B: And for Page Authority? Choose keyword.domain.com or www.domain.com/keyword? Thanks!
On-Page Optimization | | HMK-NL0 -
Absolute URLs
Hi, this is a very basic question but I want to confirm, as I remembered it was consider a good practice to use the absolute version of your links when linking to other pages of your site, not for any issue related to passing authority or PageRank, but because if someone scraps your content then they would take the links as well (as if they didn't remove them). Have the practices for internal linking with absolute or realtive URLs changed in any way? Which is the best way? absolute or relative? is there any harm for using the relative version? Relative: Absolute: [](<strong><em>http://www.cheapdomain.com/myfolder/mypage.html)[](<strong><em>http://www.cheapdomain.com/myfolder/mypage.html) [Thanks!](<strong><em>http://www.cheapdomain.com/myfolder/mypage.html)
On-Page Optimization | | andresgmontero0