Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - [email protected]
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Issue with Title of Homepage - Wordpress Platform
I can't seem to get Google to read the title of my homepage the way I would like it to be read and I have double checked in all areas of my Wordpress Platform where the title may be defaulting and all it set up correctly. I have also ensured that the "force re-write of titles and meta descriptions" have been checked through a plugin I am using to monitor SEO (Yoast)... here is an example of how Google is reading my homepage title: XYZ Company: Primary Keyword in Ontario Canada - XYZ Company I would like it to read: Primary Keyword in Ontario Canada - XYZ Company But for some reason it keeps pulling in my website title. I have always re-submitted my URL to Google but no luck with the title change. If anyone has any insight as to why this is defaulting as such I would greatly appreciate it. Thanks!
Technical SEO | | MainstreamMktg0 -
Block bad crawlers
Hi! how are you? I've been working on some of my sites, and noticed that i'm getting lots of crawls by search engines that i'm not intereted in ranking well. My question is the following: do you have a list of 'bad behaved' search engines that take lots of bandwidth and don´t send much/good traffic? If so, do you know how to block them using robots.txt? Thanks for the help! Best wishes, Ariel
Technical SEO | | arielbortz0 -
Authorship and Publisher on WordPress
I successfully enabled rel=publisher on our WordPress blog, and as a test I also enabled rel=authorship for a set of blog posts. (Tested both in Google's Rich Snippets Tester.) However, on the individual blog posts the publisher credit disappears. Is there a way to enable both to appear on blog posts?
Technical SEO | | ufmedia0 -
Rel = Canonical in Blog Posting
Hello, I keep coming back to rel=canonical issues! I noticed when I "view pagesource" that my drupal blog posting automatically creates link rel="canonical" href="/sample-blog-title" /< pattern (with the > reversed) in the source code. I'm getting a lot of Rel=Canonical warnings and double content warnings from Seomoz so I've been trying to insert link rel="canonical" href="http://www.example.com/blog/my-awesome-blog-post"< but the page won't retain the code for some reason. I'm entering the code in Plain Text, but saving the document as Full HTML. Is there a better piece of code I can put in to demonstrate that the original blog page is the original source? Thanks!
Technical SEO | | OTSEO0 -
Wordpress Category Archives
Wordpress question here. Can anyone tell me if there is an SEO advantage to creating a page filtered to show results from an individual category as opposed to simply linking to the category archive? The content is identical in both cases.
Technical SEO | | waynekolenchuk0 -
Backlink: External blog Vs. Internal blog. Which is the best?
Hi, some weeks ago a created a blog: mykeyword.wordpress.com Some one told me that it has got more trust that a "normal" www.mykeyword.com
Technical SEO | | Greenman
Is it true? So, i wrote some articles and dropped a guide (linking inside to mysite.com) to blog. My question is:
Right now i'm writing a lot of article ad i'm looking for the best channel where publish my content (post with link inside). My focus is improving quantity and quality of backlinks. Which way must i use? 1. Use my mykeyword.wordpress.com (give freshness to blog and new backlink)
2. Create ad internal blog mysite.com/blog and add article (without link?)
3. "Don't lose time" - Put new article only in external blog that will link to my site. I must manage a lot of new sites and i should increase SERP position. So, i have to choose the right way right now. Thanks 😉0 -
Thesis Vs Yoast Wordpress SEO Plugin
Hi All. I've noticed some issues between the thesis themes and Yoast's plugin, namely, thesis doesn't recognise Yoast's plugin in for title tag and meta description. Is there a way to overide this?
Technical SEO | | PerchDigital0