Cgi-bin folder
-
Is there cg-bin folder of any site related to SEO in anyways and why we always block Cgi-bin folder in robots.txt.
-
As eyepaq says - it's for holding executable "program" files. search engiens don't need access so it gets blocked. If it's empty you can even remove it.
-
No. No correlation with SEO.
However the same way you block your temp or admin folder, there is no need in letting any bot crawling your cgi-bin folder.
Cgi-bin folder holds executable file - you don't want external sources to see it anyway - it's "very" private from security reasons. You need more then knowing what's inside the folder to do harm but keeping it hidden is the wise call. Better safe then sorry.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blogger /blog Folder level redirect setup using .htaccess
We have a blog currently powered by the free blogger.com website. We have set it up as blog.example.com we wish to seti it up as example.com/blog how can we do this using .htaccess file? we understand how to update htacess, but we don't know what code we should enter to achieve what we want our website is hosted on Apache servers with plesk control panel
Technical SEO | | Direct_Ram0 -
Should Sitemaps be placed in the sub folder they reference?
I have a sitemap-index.xml file in the root. I then have several sitemaps linked to from the index in example.com/sitemaps/sitemap1.xml, example.com/sitemaps/sitemap2.xml, etc. I have seen on other sites that for example a sitemap containing blogs where the blogs are located at example.com/blog/blog1/ would be located at example.com/blog/sitemap.xml. Is it necessary to have the sitemap located in the same folder like this? I would like to have all sitemaps in a single sitemap folder for convenience but not if it will confuse search engines. My index count for URLs in some sitemaps has dropped dramatically in Google Webmaster Tools over the past month or so and I'm not sure if this is having an effect. If it matters, I have all sitemap files, including the index, listed in the robots.txt file.
Technical SEO | | Giovatto0 -
Can't get Google to Index .pdf in wp-content folder
We created an indepth case study/survey for a legal client and can't get Google to crawl the PDF which is hosted on Wordpress in the wp-content folder. It is linked to heavily from nearly all pages of the site by a global sidebar. Am I missing something obvious as to why Google won't crawl this PDF? We can't get much value from it unless it gets indexed. Any help is greatly appreciated. Thanks! Here is the PDF itself:
Technical SEO | | inboundauthority
http://www.billbonebikelaw.com/wp-content/uploads/2013/11/Whitepaper-Drivers-vs-cyclists-Floridas-Struggle-to-share-the-road.pdf Here is the page it is linked from:
http://www.billbonebikelaw.com/resources/drivers-vs-cyclists-study/0 -
URL Folders and Naming Convention Changes?
1. We’re looking for some clarification in regards to our URL structure. Currently, at our product level we have http://www.ties.com/v/a/elite-solid-black-black-tie however the parent URL is http://www.ties.com/black-ties. a. So here are the question. How much is this hurting because semantically the naming convention of this URL and weird and doesn’t follow logical patterns. In other words. Should the product page for this be http://ties.com/black-ties/elite-solid-black-tie. How bad is this hurting us? b. If we were to change the ULR structure, should we do it in phases or all at once? We don’t want to get penalized. We have well over 3,000 product pages.
Technical SEO | | Ties.com0 -
What are your thoughts on security of placing CMS-related folders in a robots.txt file?
So I was just about to add a whole heap of CMS-related folders to my robots.txt file to exclude them from search, and thought "hey, I'm publicly telling people where my admin folders are"...surely that's not right?! Should I leave them out of the robots.txt file, and hope for the best that they never get indexed? Should I use noindex meta data on every page? What are people's thoughts? Thanks, James PS. I know this is similar to lots of other discussions around meta noindex vs. robots.txt, but I'm after specific thoughts around the security aspect of listing your admin folders in a robots.txt file...
Technical SEO | | James-Distinction0 -
Blog in subfolder or folder
SEO best practices says that one should put blog in a subfolder. Like www.example,com/blog In the above case, should we say that the blog is in folder or subfolder. Actually, i have been unsure about this folder vs subfolder thing. Some examples of this would be appreciated. What is the example of a blog in a subdomain ? Thanks
Technical SEO | | seoug_20050 -
External Microsite VS Internal Folder
We would like to create either a new website or a new section of our existing website that will feature (in time) a lot of content including a forum, video training, tutorials and downloadable resources. Logistically, it would be much easier to create this in a new site (we'll call it newproduct.com) and refer people to the new site. We would, however, like to keep all of that content on our existing site for the sake of content building and SEO. Should we: Duplicate the content and use no index no follow and/or rel canonical? Host all of the content on our site and set up a vanity domain (www.newproduct.com) to point people to the deep linked area (www.mainsite.com/product/newproductinfo)? Host the content only on an external site with the occasional link back to our main site? I realize there are other options but they're mostly variants of the above. Our main objectives are to make it easy for people to get to while leveraging the new content for SEO purposes. What are the pros and cons of these different approaches? What seems to make the most sense? Thank you!
Technical SEO | | BeijerElectronics0 -
How do I use the Robots.txt "disallow" command properly for folders I don't want indexed?
Today's sitemap webinar made me think about the disallow feature, seems opposite of sitemaps, but it also seems both are kind of ignored in varying ways by the engines. I don't need help semantically, I got that part. I just can't seem to find a contemporary answer about what should be blocked using the robots.txt file. For example, I have folders containing site comps for clients that I really don't want showing up in the SERPS. Is it better to not have these folders on the domain at all? There are also security issues I've heard of that make sense, simply look at a site's robots file to see what they are hiding. It makes it easier to hunt for files when they know the directory the files are contained in. Do I concern myself with this? Another example is a folder I have for my xml sitemap generator. I imagine google isn't going to try to index this or count it as content, so do I need to add folders like this to the disallow list?
Technical SEO | | SpringMountain0