How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/
-
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can't get my preferred URL, how much does it matter?
Hi guys. I'm building a new site at the moment - seen a solid SEO opportunity for my work. I'm a producer engineer, specialising in mixing and mastering, so i'm creating a site for online mixing services. After a bit of keyword research I decided that "online mixing" was the best, most relevant and high volume term to go for. Ideally i'd like my home page to be www.onlinemixing.com (or something similar) but alas! It's been taken, as well as all the variations (like switching words, hypens etc) How much does this matter form an SEO point of view? E.g - For the search term "online mixing" would - www.onlinemixing-signalchain.co.uk be much worse than -www.onlinemixing.co.uk? Or am I sweating the small stuff? Any thoughts would be greatly appreciated. Isaac.
On-Page Optimization | | isaac6630 -
Long URL's
So I'm super new at SEO and learning a lot. I'm a small business owner and enjoy doing it myself. Are long URL's good or bad? Like this: http://www.farnorthkennel.com/german-shepherd-puppies-the-girls/long-haired-german-shepherd-puppies-lava Is that too long? The german-shepherd-puppies-the-girls is an actual page with actual content. Do those hurt me?
On-Page Optimization | | Joshlaska0 -
Categories and URL Structure - When to add a new directory?
I've been wondering this for quite awhile so I figured I should just ask. Suppose my website has 5 categories and the url structure looks like: www.mysite.com/category1/ www.mysite.com/category2/ do I also want to create a landing page for the above categories at the same URL depth as the homepage of the site? www.mysite.com/category1.html OR what about: www.mysite.com/category1/index.html Which is a better way to do this? Also, if your site began as fairly small and your 5 categories were your only other pages other than index, about, and contact pages (meaning you really had no reason to create separate directories), then as time passes, you decide to add 3 subcategory pages that would fit into a page: www.mysite.com/category1.html would you add a folder with he same name as the html page, and then rename the html file as index.html and place it into the new folder?
On-Page Optimization | | SEO-Pump.com0 -
Can you 301 redirect to a page that has other pages 301 to it?
Two years ago updated url page to include better keywords and used a 301 redirect from the old page to the new. so www.example.com/keyword-1st-generation.html now points to ... www.example.com/keyword-2nd-generation.html That moved the pages up in ranking, but now have better kw for the url, so is it okay to redirect the /keyword-2nd-geration-html to www.example.com/keyword-3rd-generation.html And what is a good length of time before removing the 1st-generation url? It's been 3 years and there is no chance of using it again. Plus, no sign of it in analytics.
On-Page Optimization | | AllIsWell0 -
Getting Google to provide a different URL in SERP
For one of my client’s sites, I have several keywords that are ranking in the top 5 positions. However, they have a high bounce rate. I believe this is because Google is delivering a different URL than the page we have optimized for the keyword. Any suggestions on ways I can get Google to present our preferred page?
On-Page Optimization | | TopFloor0 -
ECommerce URL's
This is based on a clothing retailer, eCommerce site. In an effort to reduce the length of our product names, we are considering removing terms like long-sleeve, short-sleeve, etc., but leaving that information in the URL. Now, the concern is that we would lose some traction in the SERP's if those descriptive words are left out as the product name is also our page title. Then I think keywords as broad as long-sleeve shirt wouldn't serve us well anyways. One idea we have is that the alt tag on the product image could still display the longer product name that would include long-sleeve, etc. thus having the keyword on the product page. Any ideas or suggestions? Hope this is clear. Seems redundant from a user standpoint to state long-sleeve, etc. in every product name. Thanks - your answers are always so helpful!
On-Page Optimization | | kennyrowe0 -
Does google treat all urls equal?
Sorry for the lame title, i couldn't think of a better one. I want to know if google treats this: http://www.domain.com/products/some-product-name the same as it would treat: http://www.domain.com/?products=some-product-name if not, could you tell me the differences?
On-Page Optimization | | adriandg0 -
Absolute vs relative urls
Hello, Should absolute or relative urls to be used for the internal links? I heard mixed opinions on that: One source claims that web crawlers prefer absolute urls as they are more understandable Other source points that there is no difference for web crawlers what urls are used and relative urls are shorter which reduces the size of a page. Which option is recommended? Many thanks Darius
On-Page Optimization | | LinenMe0