The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Traffic Down - May Need Outside Help
Hi Moz Community - Our website (www.motivators.com) experienced a small traffic drop in mid-March. This was followed by a steady decline in traffic through May, June and July. Please note that a site redesign went live on April 4th. Starting in mid-July, we began implementing aggressive site improvements (mostly based upon site speed), but our traffic is still down. Can anyone recommend a service or company that can look at our site and determine the root cause / more strategies for improvement? Thanks for your suggestions!
On-Page Optimization | | Motivators0 -
Understanding SEOMOZ and how to make improvements to my website
I want the same as everyone else the best position in Google. My campaign is set up as sub-domain. I am trying to get an understanding of what make the improvements to the following. What are the most important metrics and if I can work to improve them what makes those improvements. Currently I have a good position in Google for my most productive phrase but feel I need to make improvements to ensure I keep that position. I am adding regular unique content and looking for relavant quality links but it does not seem to be making much difference to the metrics below. Domain Authority 16 Domain MozRank 2.83 Domain MozTrust 0.86 External Followed Links 16 Total External Links 55 Total Links 668 Followed Linking Root Domains 8 Total Linking Root Domains 9 Linking C-Blocks 3 Subdomain MozRank 2.95 Subdomain MozTrust 1.73 External Followed Links 15 Total External Links 54 Total Links 667 Followed Linking Root Domains 8 Total Linking Root Domains 9 Which are the important metrics and can I improve my site to increase my Moz position therfore improving Google results
On-Page Optimization | | PhilSmith230 -
Seomoz.org Category and Tags practice
Hello, I have been checking seomoz sourcecode and architecture these days in order to learn and to apply it in my site but I havent managed to find "tags" at all. Just the "Posts by Categories" on the right sidebar. Is this the only way you are categorising content? In this case, the only way spiders have to find your content is via these category archive pages and the general sitemap? Thanks!
On-Page Optimization | | antorome0 -
SEOMOZ returns duplicate title and description
Hi, When SEOMOZ returns the crawl report results and it shows duplicate descriptions, is it referring to the meta description or is it referring to the actual product description? I have up to 30 products for many different sports teams. Since I have 30 products for as many teams, most of my products are the same and he only thing I did when I populated the site was to change the team name. I left the product description the same otherwise. Thanks, Don
On-Page Optimization | | ge01734000 -
Help I don't understand Rel Canonical
I'm really stuck on how to fix up Rel Canonical errors on a Wordpress site. I went in and changed all the URLs to remove the www and added / to the end. I get this message on page analysis details: <dt>Canonical URL</dt> <dd>"http://www.some-url.com.au/",</dd> <dd>"http://some-url..com.au/", and</dd> <dd>"http://some-url..com.au/"</dd> <dd>Well the first one with the www doesn't exists and the second two urls are the same! (Note that I have removed the actual URL for this post)</dd> <dd>I'm not sure how to read and fix the errors from the reports ether. The only issues I can see is that the 'Tag Value' has the www and the 'Page Title - URL' doesn't have the www.
On-Page Optimization | | zapprabbit
</dd>0 -
SEOMOZ tool - on page optmization question
Ok, I had the tool examine a page and while I got an A, it did say this..... Avoid Keyword Self-Cannibalization Easy fix <dl> <dt>Cannibalizing link</dt> <dd>"Kitchen Storage Racks", "Home Storage Racks", "Garage Storage Racks", "Wine Storage Racks", "Bicycle Storage Racks", and "wine storage racks"Explanation</dd> <dd>It's a best practice in SEO to target each keyword with a single page on your site (sometimes two if you've already achieved high rankings and are seeking a second, indented listing). To prevent engines from potentially seeing a signal that this page is not the intended ranking target and creating additional competition for your page, we suggest staying away from linking internally to another page with the target keyword(s) as the exact anchor text. Note that using modified versions is sometimes fine (for example, if this page targeted the word 'elephants', using 'baby elephants' in anchor text would be just fine).</dd> <dt>Recommendation</dt> <dd>Unless there is intent to rank multiple pages for the target keyword, it may be wise to modify the anchor text of this link so it is not an exact match.</dd> </dl> The keyword that was optimized was "storage racks". In the explanation it says that baby elephants is fine if elephants is your keyword so why isn't wine storage racks or kitchen storage racks ok what is the difference? why is mine cannibalizing but baby elephants is not? This may be very simplistic to you all but I'm learning as i go, thank you in advance
On-Page Optimization | | stacksofracks0 -
How woud you deal with Blog TAGS & CATEGORY listings that are marked a 'duplicate content' in SEOmoz campaign reports?
We're seeing "Duplicate Content" warnings / errors in some of our clients' sites for blog / event calendar tags and category listings. For example the link to http://www.aavawhistlerhotel.com/news/?category=1098 provides all event listings tagged to the category "Whistler Events". The Meta Title and Meta Description for the "Whistler Events" category is the same as another other category listing. We use Umbraco, a .NET CMS, and we're working on adding some custom programming within Umbraco to develop a unique Meta Title and Meta Description for each page using the tag and/or category and post date in each Meta field to make it more "unique". But my question is .... in the REAL WORLD will taking the time to create this programming really positively impact our overall site performance? I understand that while Google, BING, etc are constantly tweaking their algorithms as of now having duplicate content primarily means that this content won't get indexed and there won't be any really 'fatal' penalties for having this content on our site. If we don't find a way to generate unique Meta Titles and Meta Descriptions we could 'no-follow' these links (for tag and category pages) or just not use these within our blogs. I am confused about this. Any insight others have about this and recommendations on what action you would take is greatly appreciated.
On-Page Optimization | | RoyMcClean0 -
Are duplicate titles an issue for pages I don't need ranking for?
A client has a load of duplicate page titles on their site. However, to cut a long story short, most of these pages are pointless and therefore we don't need ranking for them. As such, I'm not concerned whether any of the pages with duplicate content on them are ranked or not..... unless having duplicate page titles / content on these pages could mean that other pages on the site, like the homepage, don't rank as high because of this. Do I need to worry about duplicate titles on these pages, or can I ignore duplicate content on pages that I don't want to be ranked? Hope that makes sense!
On-Page Optimization | | RiceMedia0