Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - [email protected]
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blog article cannibalizes our home page
Hello there, We're having a rather big SEO issue that I’m hoping someone here can help us with, perhaps having experienced the same thing or simply understanding what's going on. Since around June, our website's home page has lost the majority of its most important rankings. Not just dropping, but losing them entirely and all at once. We think it was self-inflicted: Almost at the same time, a blog article of ours (which we had recently updated) started ranking for almost all the same keywords. While our home page is a commercial page highlighting only our own product, the article that usurped the position is a comparison article, comparing our own solution to competitors. The reason we created that article is because we noticed a trend of Google increasingly favoring such comparison articles over dedicated product pages. But of course we didn’t plan to cannibalize our own home page with it. My question is whether anyone has experience with such a case? Is there a way to "tell"/influence Google to rank our home page again, instead of ranking that article? Thanks a lot, Pascal
Technical SEO | | Maximuxxx1 -
Should I block Map pages with robots.txt?
Hello, I have a website that was started in 1999. On the website I have map pages for each of the offices listed on my site, for which there are about 120. Each of the 120 maps is in a whole separate html page. There is no content in the page other than the map. I know all of the offices love having the map pages so I don't want to remove the pages. So, my question is would these pages with no real content be hurting the rankings of the other pages on our site? Therefore, should I block the pages with my robots.txt? Would I also have to remove these pages (in webmaster tools?) from Google for blocking by robots.txt to really work? I appreciate your feedback, thanks!
Technical SEO | | imaginex0 -
Duplicate Titles on Wordpress blog pages
Hi, I have an issue where I am getting for duplicate page titles for pages that shouldn't exist. The issue is on the blog index page's (from 0 - 16) and involves the same set of attachment_id for each page, i.e. /blog/page/10/?attachment_id=minack /blog/page/10/?attachment_id=ponyrides /blog/page/11/?attachment_id=minack /blog/page/11/?attachment_id=ponyrides There are 6 attachment_id values (and they are not ID values either) which repeat for every page on the index now what I can't work out is where those 6 links are coming from as on the actual blog index page http://www.bosinver.co.uk/blog/page/10/ there are no links to it and the links just go to blog index page and it ignores the attachment_id value. There is no sitemap.xml file either which I thought might have contained the links. Thanks
Technical SEO | | leapSEO0 -
How should we setup of a side (slightly off-topic) blog?
Our web application targets small business owners and entrepreneurs. However, the developers at our company have a lot of great content to offer the web development community and so we want to start a "behind the scenes" blog where we can discuss technical topics... JavaScript performance, web accessibility, etc. Our customers and the visitors of our website would probably not be interested this new content... So we want to be careful not to cannibalize or damage our current SEO. What are some of the major risks we should watch out for? If we put it on a subdomain, is that enough to not impact our main site SEO or introduce keyword confusion? Conversely, are there opportunities for this side blog to help the SEO and authority of our main website/domain? Thanks for the help!
Technical SEO | | Bill4Time0 -
Wordpress multisite and SEO
Newbie question. I have several keyword list I am trying to market - (My website is : http://www.soobumimphotography.com/) San Antonio wedding photographer, family photographer San Antonio sports photographer in san antonio headshot san antonio engagement photos san antonio I think it's too broad and thinking about getting separate sites for them. I know I can make urls something like this http://www.soobumimphotography.com/headshot using multisite function and planning to redirect it to www.sanantonioheadshots.com etc Is this good idea to do so? Should I get a new hosting and domain and start it all over again. Thank you
Technical SEO | | BistosAmerica0 -
Job/Blog Pages and rel=canonical
Hi, I know there are several questions and articles concerning the rel=canonical on SEOmoz, but I didn't find the answer I was looking for... We have some job pages, URLs are: /jobs and then jobs/2, jobs/3 etc.. Our blog pages follow the same: /blog, /blog2, /blog/3... Our CMS is self-produced, and every job/blog-page has the same title tag. According to SEOmoz (and the Webmaster Tools), we have a lots of duplicate title tags because of this problem. If we put the rel=canonical on each page's source code, the title tag problem will be solved for google, right? Because they will just display the /job and /blog main page. That would be great because we dont want 40 blog pages in the index. My concern (a stupid question, but I am not sure): if we put the rel=canonical on the pages, does google crawl them and index our job links? We want to keep our rankings for our job offers on pages 2-xxx. More simple: will we find our job offers on jobs/2, jobs/3... in google, if these pages have the rel=canonical on them? AND ONE MORE: does the SEOmoz bot also follow the rel=canonical and then reduce the number of duplicate title-tags in the campaigns??? Thanx........
Technical SEO | | accessKellyOCG0 -
Block Quotes and Citations for duplicate content
I've been reading about the proper use for block quotes and citations lately, and wanted to see if I was interpreting it the right way. This is what I read: http://www.pitstopmedia.com/sem/blockquote-cite-q-tags-seo So basically my question is, if I wanted to reference Amazon or another stores product reviews, could I use the block quote and citation tags around their content so it doesn't look like duplicate content? I think it would be great for my visitors, but also to the source as I am giving them credit. It would also be a good source to link to on my products pages, as I am not competing with the manufacturer for sales. I could also do this for product information right from the manufacturer. I want to do this for a contact lens site. I'd like to use Acuvue's reviews from their website, as well as some of their product descriptions. Of course I have my own user reviews and content for each product on my website, but I think some official copy could do well. Would this be the best method? Is this how Rottentomatoes.com does it? On every movie page they have 2-3 sentences from 50 or so reviews, and not much unique content of their own. Cheers, Vinnie
Technical SEO | | vforvinnie1 -
Which do you believe & why? Wordpress posts or pages?
I hear a lot of conflicting opinions regarding when to use pages over posts. If you were using wordpress mainly as a static site and less of a blog would you use pages over posts? Let's say it is a yacht review site. Would you set up most of the content (boat review pages) as pages or posts? I hear some say in that case you would want to use mostly pages and use posts for news related items. Then there are others that suggest only making the few (contact, about, privacy, terms) as pages and everything else created as posts organized into categories. Any thoughts? Also, is there a tool to find out if a website that is not yours is using a page or a post? Thanks
Technical SEO | | PEnterprises0