Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - [email protected]
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving wordpress to it's own server
Our company wants to remove wordpress from our current windows OS server at provider 1 and move it to a new server at provider 2. Godaddy handles our DNS. I would like to have it on the same domain without masking. I would like to make a DNS entry on godaddy so that our current server and our new server can use the same URL (ie sellstuff.com). But I only want the DNS to direct traffic to our current server. The goal here is to have the new server using the same URL as the old server so nothing needs to be masked once traffic is redirected with a 301 rule in the htaccess file. But no traffic outside of the 301 rule will end up going to the new server. I would then like to edit the htaccess file on our current server to redirect to the new servers IP address when someone goes to sellstuff.com/blog. Does this make since and is it possible?
Technical SEO | | larsonElectronics0 -
Getting the SEO right for blog on different server
Hi There This must be a common scenario but there's very little help on it. Right now I have: www.domain.com hosted on a Windows dedicated server. I have blog.domain.com hosted on a separate hosted Wordpress server and I use an A Record at the DNS level to make sure the sub domain works. Easy peasy! However we want to move our blog so its at www.domain.com/blog as we're definitely seeing an issue with the sub domain hosting of the blog in terms of SEO. My problem is that I cannot install WP onto the windows server, its' just not feasible as too much is going on with it, so i can;t simply redirect my blog.subdomain.com to www.domain.com/blog as it won't exist. How do I do this and maintain the SEO/link juice? Any help much appreciated!
Technical SEO | | Raptor-crew0 -
Moving a blog from Blogger
A client just setup word press. They have a ton of content on their blogger account. Today they stated they were moving the content to their blog. My knee jerk response is to leave it where it is. I am posting this question to re affirm my position. Unless there are some pro's to moving this over.
Technical SEO | | waqid0 -
Duplicate Content in Wordpress.com
Hi Mozers! I have a client with a blog on wordpress.com. http://newsfromtshirts.wordpress.com/ It just had a ranking drop because of a new Panda Update, and I know it's a Dupe Content problem. There are 3900 duplicate pages, basically because there is no use of noindex or canonical tag, so archives, categories pages are totally indexed by Google. If I could install my usual SEO plugin, that would be a piece of cake, but since Wordpress.com is a closed environment I can't. How can I put a noindex into all category, archive and author peges in wordpress.com? I think this could be done by writing a nice robot.txt, but I am not sure about the syntax I shoud use to achieve that. Thank you very much, DoMiSol Rossini
Technical SEO | | DoMiSoL0 -
Weebly Blog
Hello, Previously I asked if I should remove my blog from Weebly if I cannot change the page titles as they are being reported as duplicates. The answer was,yes and I headed to Weebly to remove it. Upon looking closer, every page and post does in fact show a different address. Now Im confused, I dont see duplicates but the campaign lists all the blog pages and states they are duplicates. I'm not sure what to do. Am I misunderstanding duplicate page? Thank you.
Technical SEO | | Gardengirl0 -
Duplicate Page Title for Wordpress
Hello, We are using WP for our blog and keep getting Dup Page Title errors for our 12 author archives pages. The title of each page is the same, but I am wondering if this is WP issue with canonicalization working properly. The most recent four pages have a linking root domain and carry some Page Authority, but the older pages do not. Is this what Rand was talking about in his Google+ whiteboard Friday talk about blog post relevancy not lasting as long as articles? Here's what it looks like. Side question, is there a reason why the SEO Moz website doesn't have a Google+ button anywhere easy to find? Thank you, Michael
Technical SEO | | MKaloud1 -
Blog Categories or Static Pages
Background: We're creating a site that for a service provider on a wordpress platform, and it will include a blog with a page for each service category. Let's say the client is a wedding photographer we'd create a page for studio portraits, events, wedding, newborns, family etc. Question: Should each category page be a stand-alone page or a blog category? As a blog category it will be updated with new content continually and automatically. However, it could cause duplicate content issues as well as keyword dilution. Research: I've done research and obviously there are conflicting opinions - skip categories altogether, categories bring traffic, problems & solutions. I'm hoping for your opinion on the matter, or direction to content on the issue that will be helpful. Blog categories yay or nay?
Technical SEO | | 5225Marketing0 -
Recommended Wordpress Plugins
Hey Seomoz'ers, What would you recommend to use as plugins for wordpress when starting a new blog?
Technical SEO | | asimahme0