How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
PA Score of 30+ to get Google Features
I was told that my pages would not be awarded a Google Feature until the Page Authority reaches 30. I had Features previously, but no longer. The PA for the URL is currently 22 https://heritageprintingcharlotte.com/products/meter-boards.php I have edited the content, built links and added schema markup to this particular page (and others) I'm looking for thoughts on the 30+ and how to improve PA. TY KJr
Moz Pro | | KevnJr0 -
Campaign Domain "change" (www. vs no www.)
Hello, I'm tracking a domain (with www.thespin.be) since months. Last week we changed things related to domain & server and now SEOmoz has 0 pages crawled. I think that's because now my main domain is without www. (thespin.be). How can I modify my campaign to reflect this "change" (that's not really a new domain..) to avoid losing all my campaign history ? Also, on Google Webmaster Tools it's like I have 2 different sites (one with www. and another without), although I've defined my favorite domain (without www.) months ago.. Google won't update its index. And I have this problem on several other sites.. Thank you very much for giving me a feedback. Regards Jean-Louis
Moz Pro | | JeanlouisSEO1 -
How to remove URLS from from crawl diagnostics blocked by robots.txt
I suddenly have a huge jump in the number of errors in crawl diagnostics and it all seems to be down to a load of URLs that should be blocked by robots.txt. These have never appeared before, how do I remove them or stop them appearing again?
Moz Pro | | SimonBond0 -
My trial ends on 2nd June. Can I cancel it on 1st or we need to initiate it before that , to not get charged ?
I need to cancel my trial. It ends on 2nd june as per seomoz. When can i cancel it exactly to avoid charges.
Moz Pro | | VarunBansal0 -
"Powered by SEOMOZ" at the bottom of White Label reports?
I'd like to upgrade to Pro Plus for the white label / own branding option. However, I have just noticed that the "white label" pdf reports still feature "Powered by SEOMOZ" at the bottom of each page. Is this a mistake? $2400 / year should be enough to remove SEOMOZ branding completely, surely?
Moz Pro | | AndieF1 -
SEOmoz bot and "noindex"
As a recent newbie to SEOmoz, I've been implementing some suggestions and doing a general tidy up. I removed URL's from our robots txt, and rolled out instead the noindex meta tag to pages we don't want indexed. But surprised to see issues that are now flagged from the last crawl by the moz bot from pages that have this meta tag? Does the SEOmoz bot not ignore this tag? Just want to make sure I've implemented it correctly, so the google bot does ignore it. Meta tag syntax is and is placed below the title tag. cheers Steve
Moz Pro | | sjr4x40 -
Idea for the Best Method of getting REAL Keyword Search Data
I've been using googles keyword tools to get keyword search data, but I'm pretty sceptical of some of the results it throws out on number of searches. On a number of keywords - particularly when you get to longtail, I've seen results which cannot possibly be right. So - I am trying to find out a way of getting REAL data I can trust. Here are my thoughts on a possible method of getting this key data:
Moz Pro | | James77
1/. Firstly I am taking on the assumption that the adwords campaign monitoring statistics are more reliable and accurate than the keyword research tools - correct me if I am wrong! 2/. Then take a load of keywords you wish to monitor and create didfferent ads for them based of EXACT match. 3/. Make sure non of your adverts conflict with each other, so you may have to turn off any other campaigns you have running. 4/. Put you bids and available spend as high as possible, so that ideally you will your ads will always be shown for every search match, and ideally always come in No1 spot on the ads. IE your aim is to create a neutral environment where your ads are shown for every search match and appear in the exact same position for each search match. Run the campaign for a period of time long enough to be confident that you have enough search data. From this you should have the vital data on keyword searches done on your exact match keywords. 5/. Repeat the test, but this time use phrase matches and again make sure there are no conflicts. 6/. Repeat again with broad match - this would require very careful implimentation not to have any conflicts and you would likely need to make heavy use of the "Not Include" keywords. What are your thoughts on the above process? - Any flaws, or other better solutions? Obviously one key thing with doing this is you need to have to be prepared to have a decent budget to get this data - but it won't be wasted as you will also be getting the adwords traffic. Thanks0 -
How to get seomoz to re-crawl a site?
I had a lot of duplicate content issues and have fixed all the other warnings. I want to check the site again.
Moz Pro | | adamzski0