Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Https Implementation - Weird Redirection After Putting 's' in http://
Hi Mozers, I have come across some websites with their https version going to a totally different website. For example, http://www.samplesite1.com will load fine but when the protocol is changed to https (https://www.samplesite1.com) it will go to a total different domain say, https://www.samplesite2.com How does this happen, in technical sense? In the warning from browser, it says the the security certificate is from the other website but I would like to understand how this happens and how it impacts SEO. I seem to be not able to understand the relationship of this error and SEO impact. Thanks in advance for your response. Malika
Web Design | | Malika10 -
Using Multiple links/names for the same product?
I am being asked to change these product links on the home page: Home/Condo
Web Design | | RoxBrock
Watercraft/Boat to Home
Condo
Watercraft
Boat (Along with several other product links) How does this affect the customer experience/usability, and SEO? Is it a good idea or is it confusing? Thank you.0 -
May I know what's the "CSS Class" here?
ul.nav a#homelink { text-indent: -9999px; background: url('http://www.elegantthemes.com/preview/MyProduct/wp-content/themes/MyProduct/images/home-icon.png') no-repeat center 6px; padding: 0px; } ul.nav a#homelink span { display: block; padding: 5px 9px 8px 5px; width: 16px; } I'm having problems in adding back the "house" icon instead of a "Home" text in my menu. I don't what "CSS class" should I use to be added in the Home menu button (Appearance>Menu). I'm currently using a "MyProducts" Wordpress theme by ElegantThemes. Thanks in advance!
Web Design | | esiow20131 -
How does using a CMS (i.e. Wordpress/Drupal) affect backlinks and SEO?
So I need to build a website with over 100 pages in it. Elements of the design will probably be moved around and or tested so I need to use a CMS. It's pretty much a review site so while the content will remain static I'd like to employ A/B testing to mess with conversion rates. Wordpress has a plugin for that even. So I'm just wondering, since CMS pages are pretty much created on spot and not retrieved from a library, how this affects backlinks and anchor text? How exactly does the external website point to yours if the URL is dynamically generated? Or am I misunderstanding something? Please recommend any extra resources as well if you can.
Web Design | | seochump0 -
Word Press Seo Errors/ Questions
Hi my name is Tina I am new here I hope you guys can help me out. I thought building my new site with Word Press was going to simplify things, however I have a ton of errors, and I am not sure what they are, or how to fix them. I am hoping someone could share with me a solution for these errors. I have 28 rel=canonical errors, I am not sure what this means, I understand it to mean my pages are similar, and this is to set a heirarchy between my pages. Please correct me if I am wrong. If I am correct would this be necessary to add if my main keyword was "widgets" and my home page was optimized for "widgets" and my next page was "blue widgets" and so on. While my pages are similar they are all optimized for different versions of my main keyword some using long tail keywords. Do you know of a plugin that can help solve this problem? Also does anyone have a plugin they recommend for G+ my G+ authorship verification is causing an error as well? I am using Head Space 2 I have used this seo plugin numerous times with great success it has been my favorite seo plugin. However, we have a portfolio that shows our clients websites, and on those pages Head Space will not let me enter a description tag. What plug in do you guys recommend with more control over each page? Another interesting issue is on one of our pages I optimized it for our Canadian clients, and now every page has been listed in Google.ca for the keywords it should have on Google.com. We are listed on Google maps, verified in Google places, and our address is on the site so they know we're from the USA however, the majority of our keywords are only listed in Google.ca. We're on page one for all of them, we are in the top three on most of them so that's not bad, but we want to be listed in Google.com as well. Any suggestions on this?
Web Design | | TinaGammon1 -
Given the lastest Google update, should I rewrite my Flash site or try to present an alternative HTML/CSS site?
I have a site that was created using Flash. The reasoning behind this was, at the time, that I didn't care if the site ranked or not (portfolio site). Now I would like to drive traffic to the site from SE's. Given the Penguin update, should I rewrite my Flash site in HTML/CSS or present an alternative site for bots and browsers that don't support Flash? My concern is that by presenting an alternative site to bots and non Flash supporting browsers that the SE's will see potentially see this as cloaking. Thoughts and advice would be much appreciated.
Web Design | | mj7750 -
Setup of three major retail sites.. need advice.
I recently have taken a new position responsible for three large national retail sites which are all owned by one parent organization. Through a series of acquisitions, these three major brands have been brought under one umbrella and a brand consolidation is likely not to happen within the next 2-4 years. I have a number of questions I’m hoping to get some feedback on, but first a little more background is necessary. A year ago (before my time) the three sites were over-hauled, but were designed to use one common custom CMS and all of the navigation and nearly all the content is the same (with some exceptions, such as tags, url, etc.). All of the brands have identical products and services; however, each one services a different demographic in the US. The design was intended for ease of management, but is terrible for seo. Additionally, without the geographic reference, they all compete for the same keywords. They have now begun a very large ecommerce project utilizing an ATG platform. The initial direction is to use one platform for all three brands, but keep them on separate domains and with the use of basic switching, replace nominal content such as logos and references of the brands for each of the domains. I’m concerned with this approach and would like to hear your feedback.. When optimizing a page for one keyword set, are they likely to be filtered due to dup content? The argument that management has is that all three current sites rank very well for one keyword on all three sites. They feel it won’t be an issue due to this. One option, that is currently still available, is to tri-band one ecommerce site, but it would have to be on an entirely new domain. The other three domains are very well established and are PR6s. Management, and even I, is afraid to abandon these other domains, but having a single domain would allow us to have unique content and really leverage all efforts to one domain. Thoughts? Any knowledge or thoughts what kind of impact having three domains on one ATG platform will be? Thanks much! John If you feel it will help, please message me and I can share the urls... Also, how would you handle a company blog in this case?
Web Design | | kavaliauskas0 -
Need advice on diplaying content for Search
Hi every body, I am doing landing page redesign(s). Does any body know or can refer a content carousel that can rotate video and pictures? The "site with images" search option result is a compelling reason to showcase pictures if your space competeiveness (showroom, merchandise, etc) can be improved with a strong image presence. here is my main landing page http://www.shearerpainting.com I know there is alot of stuff, and confusing call to action, but I am looking for strategies to clean it up, clear fous on action (get bid, learn more), but allow users to see that they can dig for more content.
Web Design | | johnshearer0