Search engines have been blocked by robots.txt., how do I find and fix it?
-
My client site royaloakshomesfl.com is coming up in my dashboard as having Search engines have been blocked by robots.txt, only I have no idea where to find it and fix the problem. Please help! I do have access to webmaster tools and this site is a WP site, if that helps.
-
Here's the screen in Wordpress to check to see if you've blocked search engines through WP.
-
Ah, I see. Thanks!!
-
nothing comes back when I run this
site:www.royaloakshomesfl.com
So looks like its not indexed
-
When I search their brand name, they pop up in Google, so they must be indexed somehow, no?
-
I would do as Lavellester suggest, then i would then i would re-submit as i can see you are not in either index.
-
You can see it here:
http://royaloakshomesfl.com/robots.txt
So its in the webroot of your hosting. You can change it via FTP to and change it to the following to grant full access until you have figured out robot access rights. I think you need at least one Disallow: line without the / slash in your existing configuration to make it work.
User-agent: * Disallow:
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots file set up
The robots file looks like it has been set up in a very messy way.
Technical SEO | | mcwork
I understand the # will comment out a line, does this mean the sitemap would
not be picked up?
Disallow: /js/ should this be allowed like /*.js$
Disallow: /media/wysiwyg/ - this seems to be causing alerts in webmaster tools as it can not access
the images within.
Can anyone help me clean this up please #Sitemap: https://examplesite.com/sitemap.xml Crawlers Setup User-agent: *
Crawl-delay: 10 Allowable Index Mind that Allow is not an official standard Allow: /index.php/blog/
Allow: /catalog/seo_sitemap/category/ Allow: /catalogsearch/result/ Allow: /media/catalog/ Directories Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/ Disallow: /media/ Disallow: /media/captcha/ Disallow: /media/catalog/ #Disallow: /media/css/
#Disallow: /media/css_secure/
Disallow: /media/customer/
Disallow: /media/dhl/
Disallow: /media/downloadable/
Disallow: /media/import/
#Disallow: /media/js/
Disallow: /media/pdf/
Disallow: /media/sales/
Disallow: /media/tmp/
Disallow: /media/wysiwyg/
Disallow: /media/xmlconnect/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
#Disallow: /skin/
Disallow: /stats/
Disallow: /var/ Paths (clean URLs) Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product/gallery/
Disallow: */catalog/product/upload/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/ Files Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Disallow: /get.php # Magento 1.5+ Paths (no clean URLs) #Disallow: /.js$
#Disallow: /.css$
Disallow: /.php$
Disallow: /?SID=
Disallow: /rss*
Disallow: /*PHPSESSID Disallow: /:
Disallow: /😘 User-agent: Fatbot
Disallow: / User-agent: TwengaBot-2.0
Disallow: /0 -
Fix or Block Webmaster Tools URL Errors Not Found Linked from a certain domain?
RE: Webmaster Tool "Not Found" URL Errors are strange links from webstatsdomain.com Should I continue to fix 404 errors for strange links from a website called webstatsdomain.com or is there a way to ask Google Webmaster Tools to ignore them? Most of Webmaster Tools "URL Not Found errors" I find for our website are from this domain. They refer to pages that never existed. For example, one was to www.mydomain.com/virtual. Thanks for your help.
Technical SEO | | zharriet0 -
The search button on my site no longer works. Any fixes?
I have a wordpress site (can be found at theFlooringGirl.com). On all the pages, there is a search button so you can search for words on the site, but it no longer seems to work. My original webdesigner put it in, and I can't seem where to find it/where to edit/fix it. I finally found something that mentions "search" in the widget section and it is in the tool bar, but I can't find where/how to edit/fix it. BTW, I do have two other smaller sites that have this button and it still works there...so if I could find where to edit, I could theoretically copy and paste the code...if I knew where). Any advice? Thx.
Technical SEO | | Jborgueta0 -
Explain this search result
Hi folks, I came across a strange search result. Search on Google Australia for "income portfolio". http://www.google.com.au/search?sourceid=chrome&ie=UTF-8&q=income+portfolio See the first result? It's a login page. How is that search result showing? And in position #1! Where is it getting its title and descriptions tags from? Does Google have a way to somehow see what is behind the login? Appreciate your thought.
Technical SEO | | scotennis0 -
Anchor text with class atttribute followed by search engines?
Hi folks, Is an anchor containing a class attribute followed by the search engines? Viewed the "text only" version of the cached page in Google and the link is not listed so not sure if it is followed for indexing purposes The code: Reports Actually the link was recognized as an internal link using the SEOMOZ toolbar and also was listed as an internal link using A1 Analyzer. Any thoughts welcomed!! Finn
Technical SEO | | insite3600 -
Do the engine spiders see this javascript?
http://www.drsfostersmith.com/general.cfm?gid=259 I'm looking at building something like these banners, and wondering if the engines a) see b) value links like these in the drop-down selector? I guess I could test it but wondering if anyone else has before I do it.
Technical SEO | | ATShock0 -
Robots.txt file getting a 500 error - is this a problem?
Hello all! While doing some routine health checks on a few of our client sites, I spotted that a new client of ours - who's website was not designed built by us - is returning a 500 internal server error when I try to look at the robots.txt file. As we don't host / maintain their site, I would have to go through their head office to get this changed, which isn't a problem but I just wanted to check whether this error will actually be having a negative effect on their site / whether there's a benefit to getting this changed? Thanks in advance!
Technical SEO | | themegroup0 -
Up-to-date list of search engine bot user agents
Does anyone know of an up-to-date-list of search engine bot user agents? Thanks.
Technical SEO | | JoeAmadon0