How to Hide Directories in Search?
-
I noticed bad 404 error links in Google Webmaster Tools and they were pointing to directories that do not have an actual page, but hold information.
Ex: there are links pointing to our PDF folder which holds all of our pdf documents. If i type in , example.com/pdf/ it brings up a unformated webpage that displays all of our PDF links.
How do I prevent this from happening. Right now I am blocking these in my robots.txt file, but if i type them in, they still appear.
Or should I not worry about this?
-
Yes, a visit to example.com/dir should now return a 404 error (if you haven't done any redirecting/canonicalizing). This will increase your 404 count in Web Master tools but it's far preferable to the alternative. If you're not redirecting the robots.txt will eventually work and hopefully the links will just fall out of WMT.
-
My hosting company turned off directory browsing and now everything is how it should be. So to my understanding, if the server sees a file that does not have a index file, it should not be view able and should be forbidden. This shoujld not affect us from an SEO standpoint should it? My hosting company said they disabled all directories in our site, however everything still works, except for the forbidden file directories.
-
Basically it shouldn't really have an affect; those unformatted file listings are literally the web server automatically saying 'here's the files that are in this folder', there's no meta tags, description, on page elements, etc.
If you have these pages and they're ranking well, you generally don't want them to be. The automatic file browsing pages don't have your name, your company, etc. in them, and they're generally pretty ugly. They also theoretically could be 'stealing' juice from your 'real' pages, if your internal structure isn't flowing relevance properly.
Basically what I'm saying is that if these pages are having some kind of SEO effect, you probably don't want them to be since they're so basic.
Also I can't overstate the security concerns that directory browsing might be introducing. If someone can directory browse to where your code lives (.php, .aspx.vb, whatever) they may be able to read it. Code sometimes has important things like logins, passwords, merchant account ids, etc. in it that you definitely don't want people reading.
-
Agreed with Valerie that step 1 is to turn off those directory listing pages - that can be a security issue and you don't necessarily want people to see/access the whole list. Also, make doubly sure you don't have any internal links to that directory (Google crawled it somehow).
Generally, Robots.txt should prevent crawling, but it's not foolproof, and it's pretty bad about removing pages once they're indexed. If you can block the page from browsing and return a 404 for the root page, that should be fine. The other option would be to have the page removed in Google Webmaster Tools. You could request removal for the entire folder, but I'm guessing that you may want the actual PDFs indexed.
-
Will turning of directory browsing affect Search for all directories?
-
I really don't want to 301 redirect them as they are just holding files. This is happening with my includes file too. that holds our header, footer, navigation etc. I can check with our hosting company to find out.
-
I'd create an index.html for the directory, and then redirect it somewhere. This way, you're capturing the inbound links and then rescuing some of the inbound juice.
Otherwise, you can also check out this post for more info on other solutions and modifying your htaccess file to prevent the directory view - http://perishablepress.com/better-default-directory-views-with-htaccess/
-
Blocking it in robots.txt will work to hide it from search engines.
If you want to hide it from users or people to who type in the url, you can simply drop a blank "index.html" in the /pdf folder.
-
I would suggest 301'ing them to their /index.htm or /pdf.htm equivalents. If you don't know, a 301 is a signal to a web browser (or search crawler) saying "this page has permanently moved, please go to (otherpage.htm) instead".
Here's a good SEOMoz article explaining it a bit more:
http://www.seomoz.org/learn-seo/redirection
What might be more of a concern, is it sounds like your web server has directory browsing enabled. This could be a security issue (depending on your web server setup). Generally you don't want to expose directories if you don't have to because it gives a potential attacker insight into your system setup. Here's an example how to do it in Apache:
www.camelrichard.org/topics/Apache/Turn_OffDirectoryBrowsing
And IIS:
technet.microsoft.com/en-us/library/cc731109(v=ws.10).aspx
If you like I can confirm if you have open directories if you give me the link, either here or through private message.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why are these pages showing up as 404 in Search Console?
This page is showing up as a 404 in Google Search console- https://www.wolfautomation.com/blog/autonics/ It shows it has been linked from these pages- https://www.wolfautomation.com/blog/raffel/ https://www.wolfautomation.com/blog/new-temperature-controllers-from-autonics/ https://www.wolfautomation.com/blog/ge-industrial/ https://www.wolfautomation.com/blog/temp-controller/ https://www.wolfautomation.com/blog/tx4s/ I never created this page, I don't want this page but it keeps showing up. The problem is the link isn't found on those pages anywhere so I can't delete it. What am I missing? How can I get rid of it?
Intermediate & Advanced SEO | | Tylerj0 -
How much SEO damage would it do having a subdomain site rather directory site?
Hi all! With a coleague we were arguing about what is better: Having a subdomain or a directory.
Intermediate & Advanced SEO | | Gaston Riera
Let me explain some more, this is about the cases: Having a multi-language site: Where en.domain.com or es.domain.com rather than domain.com/en/ or domain.com/es/ Having a Mobile and desktop version: m.domain.com or domain.com rather than domain.com/m or just domain.com. Having multiple location websites, you might figure. The dicussion started with me saying: Its better to have a directory site.
And my coleague said: Its better to have a subdomain site. Some of the reasons that he said is that big companies (such as wordpress) are doing that. And that's better for the business.
My reasons are fully based on this post from Rand Fishkin: Subdomains vs. Subfolders, Rel Canonical vs. 301, and How to Structure Links for SEO - Whiteboard Friday So, what does the community have to say about this?
Who should win this argue? GR.0 -
Directories that Redirect - Do They Pass Link Juice?
I did some searching before asking but could not quite find what I was looking for. There are valid directories out there that provide business as well as links that provide SEO value. My question is whether or not having a redirect in place negates passing any link juice. When I use Open Site Explorer for Old Monterey Inn, this directory (CABBI) does not show up on their list. However, their website dropped from Google Analytics altogether for some time because of an issue in how they built their site. Their "fix" is this redirect which was integrated a short time ago. I do see traffic in Google Analytics now but wonder about the link juice. Example: <a href="[/redirect?type=website&inn=34211&url=http%3A%2F%2Fwww.oldmontereyinn.com](view-source:https://www.cabbi.com/redirect?type=website&inn=34211&url=http%3A%2F%2Fwww.oldmontereyinn.com)" target="<a class="attribute-value">_blank</a>">www.oldmontereyinn.coma>p> What say you? Thanks to anyone that responds.
Intermediate & Advanced SEO | | ColoradoMarketingTeam0 -
Adding some unique text to search
I do not know how to easily explain that but I will try 🙂 We are running the yellow pages in our country, where most (all) of the companies are listed. Each company has their keywords that are used in our search. All of those search keywords are indexed by Google and it goes something like that domain.com/search/keyword So far this page only lists the companies that match the search query. So that's not much content. I'm thinking about adding a sentence that would go something like this:
Intermediate & Advanced SEO | | FCRMediaLietuva
The search query lists the companies of our country that do KEYWORD. So far COUNT was found. That seems pretty short and I am not sure if that does a lot of help for Google. But I cannot think of more text. Any ideas? Or should I give up on the idea of adding anything at all?0 -
Search traffic down 30% this month
Our search traffic has been growing at a steady clip for the last year but is down about 30% this month. As part of a redesign, we've repurposed our home page (blog.getvero.com). Rather than serve as a feed of recent posts, it's now an email signup page. We created a new page (blog.getvero.com/posts/) to display new posts. I think this is likely the reason for the drop in search traffic but I'm frustrated that it's losing us thousands of visitors per month. A few questions: 1. How long will it take to recover from this? 2. Is there anything we can do to speed up the recovery process? 3. Why are some of our best performing posts seeing less search traffic even though the URL hasn't changed? Any help is greatly appreciated.
Intermediate & Advanced SEO | | Nobody16116983020420 -
Sudden drop in ranking for major search terms
Site bumpbabyandbeyond com au. Help! We have been operating for six years and had steadily built up our ranking for major terms like maternity clothes, maternity clothing, maternity wear, reaching highs of 6, 8 and 10 respectively for AU sites AU wide about six months ago that we have steadily maintained. All of a sudden we have dropped away. A week ago I noticed we had dropped from 6 to 12 for maternity clothes. This morning we are 21! I can't see any obvious reason for this, but believe the eCommerce module of our inventory/pos software has had a recent update - I'm awaiting answers on this. We haven't actively had anyone link building or working on SEO after being badly bitten and shelling out a small fortune for an AU company to do very little over six months - rankings improved rapidly when I sacked them and did some on page minor work myself. But I don't have the time or knowledge to look after the seo, and am on the hunt for reputable white hat assistance. Is there anything obviously wrong that I need to fix ASAP? Any help would be much appreciated 🙂
Intermediate & Advanced SEO | | catfree0 -
Language Attribute - does changing it make a difference to SEO and Search?
I am an SEO newbie, but learning fast. 🙂 I am based in London, UK and have a website: www.twofourseven.co.uk. I noticed that the language attribute was set to 'en-US'. I work in London as well in international locations in the Middle East and Asia. Thinking of this I wanted to ask the experts if given that I am based in the UK, would changing the language attribute make a difference to search results? If so, would 'en' be better than 'en-GB', which might be too specific? Thanks in advance!
Intermediate & Advanced SEO | | twofourseven0 -
Strange issue with video search results...
Hi all, Got a bit of a weird problem that I can't work out. I've got a page that contains a video. The SERP for one keyword has the video appearing directly in the search listing like a video rich snippet / schema. Do not want. This rich snippet style video result only appears when the page is found for this one keyword, and no other. How do I stop google displaying the page like this? Why is it only displayed like this for one keyword and no others? The video is a YouTube video and is embedded in the page. Nothing fancy is going on with the code. Any ideas? I'm stumped.
Intermediate & Advanced SEO | | WillQ0