Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - [email protected]
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Page Titles For Paginated Topics In Blog
Hello, I've just run a site audit and it has come up with a duplicate title tag issue for the topics section of our blog. For example it is flagging that the following have the same page title. https://blog.companyname.com/topic/topic-name https://blog.companyname.com/topic/topic-name/page/2 How significant is this as an SEO issue and what are the ways we can go about fixing this? I look forward to any suggestions and guidance that can be provided. Thanks, John
Technical SEO | | SEOCT1 -
Www to non www on a .com/blog url
hi guys, I have had to reset my site from www to non-www. via htacces and this worked out just fine.However, the /blog WordPress section will not redirect to the non-www. I have changed the config.php to non-www. However, the /blog WordPress section will not redirect to the non-www. I have changed the config.php to non-www. Does anyone have an idea as to what I need to do to force the non-www in a folder installed blog http://5starweddingdirectory.com/ http://www.5starweddingdirectory.com/blog/ Regards T
Technical SEO | | Taiger0 -
Problem with Wordpress RSS feed and Feedburner
Just discovered a problem with my company site's RSS feed. I'm a bit embarrassed to ask, but I thought someone in the community might have encountered this -- and I cannot figure it out for the life of me! We had redirected our Wordpress feed to Feedburner. We publish at least once per week, but no posts after March 18 are in the feed: http://feeds.feedburner.com/TheClineGroup The standard (Wordpress) RSS feed page does not load: http://theclinegroup.com/feed/ Of course, I deactivated all plug-ins to see if one of them was the issue, but the problem(s) still existed. Thanks so much for any assistance!
Technical SEO | | SamuelScott0 -
Htaccess help... I moved my blog from a seperate domain to newdomain.com/blog
Hi, I need help with my htaccess file, I've been told. I moved a blog i had hosted somewhere else to a directory on my ecommerce site. I was told i would need to write something to go in the htaccess file so the sites would not become duplicate content, but I'm a novice and have no idea how to write that code. blog moved from www.whosyourmoondoggie.com to www.moondoggieinc.com/blog Please help, or direct me to the right tutorial 🙂 Thanks! KristyO
Technical SEO | | KristyO0 -
E-Commerce site and blogs
We have e-Commerce site and an official blog to give advice about our products. This blog exists under our domain. Usually we build links directly to our site. Recently our ranking started going down. Also, we have been experiencing backlash for spam based on our link building (we are working on this, including a change of staff,but we cannot be sure that this will not happen again). This backlash has come through our social networking outlets (Facebook) in the form of very negative posts to our pages. One of our "SEOs" has devised a plan to use secondary blogs which we would start building links for. This blog would contain links back to our website. The idea is that the blog acts as a gate in a sense, in this way backlash is either posted on the blog or is directed at the blog. Also, we would be attempting to raise the page authority of these secondary blogs so in essence they act as high page authority links back to our website. The concern is that these secondary blogs may undermine the legitimacy of the official primary blog, which is still in its early stages as far as ranking and authority goes. Also, we are concerned that this technique would further undermine the legitimacy of the website itself by creating a larger "spam-like" presence, since visitors may see through the use of the secondary link through blogs.
Technical SEO | | ctam0 -
Leaving Comments on blogs when html is removed
I found the following blog. It is pagerank 5 do follow http://www.unssc.org/web1/programmes/rcs/cca_undaf_training_material/teamrcs/forumdetail.asp?ID=32 If you attempt to leave a comment with html, the html is removed. There is a button which allows you to leave a comment but if you do it gets redirected to the domain of the blog not your site. However there are still people leaving links with the url of the intended site. As late as today. look at this comment
Technical SEO | | mickey11
Comment posted by : Alex on 09/09/2011 I love to se percorsi on this site very often How is this done, if anyone knows I got the code done to this your keywords The important part being mce_real_href0 -
What should be noindexed on a Wordpress blog?
I know this can be a "it depends" answer so I'll try to explain. Qualifications on your answers would be great. I use the Wordpress architecture for myself and clients on sites and blogs. Almost every business site we create has a blog and I'm always working to improve results on them. My strategy has been the following: Categories: General, main content types, general keywords. Index, follow Tags: Very specific, post specific, may only be used once for one post. My categories have descriptions that are displayed on the category pages with excerpts. Tags rarely have a description but are displayed with excerpts on the page. My idea has been to index the categories to crawl the content and they have unique content by showing the category description. Tags shouldn't be archived because they may be all over the place and may have only 1 post with no tag description. I'm trying to reduce duplicate content but I don't want to limit results for my clients and myself. Should I set tags to noindex, follow or should I have them indexed? The only thing I'm thinking with having the tags indexed is that I may be able to get additional traffic through the more specific tags (i.e. tag = meta tags, category = SEO).
Technical SEO | | JaredDetroit0