Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - [email protected]
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Structured markup for wordpress
Hello, I am having problems with marking up my WP posts. I used the All-in-One which seems to be the most user-friendly, except when I denote the aspects of the "article" and update, the markup shows up as a box at the bottom of the post (even though the info is in the text). How do I mark these up for Google without having the unseemly box at the bottom? Thanks so much in advance for any help! Btw, I am not altogether comfortable just yet on manual schematic markup (if you have a really basic manual markup that will let me do so across various platforms, I would also appreciate the recommendation). Thanks!
Technical SEO | | lfrazer1 -
How do I redirect the Author archive page in Wordpress?
If you do a search for my name on Google, the first result is the author archive page of my Wordpress blog. I would like to redirect the author page to my "about me" page but cannot add a 301 as the author page is created dynamically in Wordpress. Anyone know how I can do this?
Technical SEO | | richdan0 -
Problem with Wordpress RSS feed and Feedburner
Just discovered a problem with my company site's RSS feed. I'm a bit embarrassed to ask, but I thought someone in the community might have encountered this -- and I cannot figure it out for the life of me! We had redirected our Wordpress feed to Feedburner. We publish at least once per week, but no posts after March 18 are in the feed: http://feeds.feedburner.com/TheClineGroup The standard (Wordpress) RSS feed page does not load: http://theclinegroup.com/feed/ Of course, I deactivated all plug-ins to see if one of them was the issue, but the problem(s) still existed. Thanks so much for any assistance!
Technical SEO | | SamuelScott0 -
Small blog needs paid SEO help
I looked at Moz's recommended companies but they are all for huge blogs and websites. I cannot afford those fees. My blog is small but growing and was hit by a Google Panda tweak back in July. My organic traffic disappeared overnight and although I've been working hard to improve my blog, I still get no organic traffic after being on the first page. I would like to pay someone to look at my site and give me help but I cannot pay what these huge companies are asking for. Any advice?
Technical SEO | | blogger20130 -
Wordpress Woocomerce Recommended SEO URL structure
Hi Mozzers ! Thanks for looking. I have a new shop in development (http://www.vintageheirloom.biz), I'm now using WordPress & Woocommerce. I've asked Woocommerce whether it is possible to remove the 'shop' and 'product-category' categories. They say it is, but it isn't recommended, it can slow site speed & create possible duplicate pages. I'm wondering what seasoned SEO experts opinions are on my particular structure? I've heard that a flat structure is recommended, but ecommerce shops as I understand pose their own issues, so any feedback would be appreciated.. Here's some URL examples: http://vintageheirloom.biz/shop/bags/ - this for the category bags http://vintageheirloom.biz/product-category/bags/shoulder-bags/ - this for shoulder bags a child of bags category http://vintageheirloom.biz/shop/2-55-bags/vintage-chanel-caviar-skin-2-55-bag/ - a product The last URL contains the category 2-55 bags. The products name also includes the phrases 2-55 bag. Should this level of repetition be avoided or is it best to keep the whole phrase 'vintage-chanel-caviar-skin-2-55-bag/' for SEO purposes? Thanks for any help you can give me around this issue! Kevin
Technical SEO | | well-its-1-louder0 -
Does Having Links on this blog Hurt or Help?
Hello, I created a wordpress blog a while back, http://plastic-bins.com/ If you go into one of the pages on the blog (for example: http://plastic-bins.com/plastic-shelf-bins/ ) you will notice a link after the text/content telling you where you can purchase the stuff that is talked about on that page. There is one link back to the e-commerce site on every category page on the blog. Does anyone know if these links will help me or hurt me in terms of ranking in the SERPS? Thanks
Technical SEO | | Prime850 -
Redirect blogspot blog ?
Hi We have an ecom site selling childrens clothing. We startet up with a blogspot blog due to the lack of that option from our provider. I have a few links in to the blogspot blog and realized today that the PA is 39 - the PA of homepage of ecom site is 40. Ive lately been using the ..../blog on my ecom site. The functionality so far of it is so bad that i am considering using the blogspot blog again. How can i best utilize the strength of our blog at blogspot. Just keep posting w anchor text to relevant pages our main site? Is redirecting a possibility at all ? Dan Lærum
Technical SEO | | danlae0 -
Block a sub-domain from being indexed
This is a pretty quick and simple (i'm hoping) question. What is the best way to completely block a sub domain from getting indexed from all search engines? One item i cannot use is the meta "no follow" tag. Thanks! - Kyle
Technical SEO | | kchandler0