Blocked By Meta Robots
-
Hi
I logged in the other day to find that over night I received 8347 notices saying certain pages are being kept out of the search engine indexes by meta-robots.
I have not changed my robots.txt in years and I certainly didn't block Google from visiting those pages.
Is this a fault on Roger Mozbot behalf? Or is there a bot preventing 8000+ pages being indexed?
Is there a way to find out what meta-robot is doing this and where? And how I can get rid of it?
I usually rank between #3 and #5 for the term 'sex toys' on google.com.au, but I now rank #7 to #9 so it would seem some of my pages/content is being blocked.
My website is www.theloveshop.com THIS IS AN ADULT TOYS SITE.
There is no porn videos or anything like that on it, but just in case you don't wish to look at sex toys or are around kids I thought I would mention it.
Blake
-
Hi Nakul
Thank you very much for the suggestions and I will start checking the code on those pages today. Thanks for heads up on the .com.au site as well I will be changing the hosting today to Aus, and thanks to you that site may get pushed up the rankings for a highly competitive term that I have been fighting to get for years
As for following my competitors links, I have been doing that for awhile and it would seem most of them either buy their links or have them on pages that are irrelevant to our niche, one of my competitors has even acquired more links than SEOmoz.com but has no were near the content. In saying that I am by no means a pure white hat SEO, more of a grey hat if such a thing exists, simply because I find it hard to get links to an adult site without it being spammy of some sorts. So I have started writing articles and linking to products mentioned, but trying to make the articles interesting/funny at the same time. Hopefully soon I will start to see some improvement.
Again thank you for your help and I will let you know how I go with those pages blocked by bots.
Blake
-
The meta robots block sounds like it's a block from your CMS or something in your header include file or something. I would check on the pages that show as blocked to see if there's any noindex sort of code in those pages. Other then that, shift in rankings could be based on your competition optimizing for the same set of keywords. I would recommend to spend some time analyzing the backlinks of the competition and see if you can gain some of those kinds of links.
On a side note: It also looks like you own both the .com and the .com.au site. The .com is hosted in AU and the .com.au is hosted in USA. Did you intend to do it the other way around ? I would imagine your rankings for the .com.au get better if it was hosted in Australia instead of USA.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will German meta data help an English USA-based site get found in Germany?
I think I already know the answer to this, but I'd love a second opinion. If you add German meta data to some web pages will it help those pages turn up in search results in Germany?I'm assuming not. It's an English, USA based site, doing well here for English searches - and I don't want to mess that up. A German site is being planned, but this client is hoping the English site could be found in the meantime. ~Caro
On-Page Optimization | | Caro-O0 -
Duplicate meta data for 301 redirected items
Google webmaster tools tells me that I have duplicate title tags for two items. They have the same product ID. When there is a change in SEO URL, the system does an automatic 301 redirect of the first item towards the second item, like in the example below:
On-Page Optimization | | Madlena
22"x28" White Foam Board School Project Kit|Geographics.com
/p48998/foam-board-school-project-kit-22-x-28/product_info.html /p48998/white-foam-board-school-project-kit/product_info.html What are we doing wrong that Google asks for different meta data for a 301 redirect? Thanks!0 -
Should I include a meta description for my tags on posts
Hi Our crawl report detects that we have not put in a meta description for any of the tags we have used on our blog. (total tag words we have used is around 70). We have however, put a full description of each of our categories, do we really need to put a description for each of the tags we use? Will this really help us with SEO? Also can tags on posts be the same as the keyword your trying to rank the post/article about? For further information We are using Yoast.
On-Page Optimization | | lethalmarketing0 -
Selling Products with a similar meta description
Wondering if anyone can help when selling similar products with very similar meta description and product descriptions in general. Have around 500 products - a lot of products have around 10-20 products which are very similar only different is sizes and a maybe a few lines of text if that. Is this a problem in search engines? How does other ecommerce stores selling similar products solve this problem...
On-Page Optimization | | royRR0 -
Wordpress Duplicate Meta Title Issue
Hi, Google webmasters tools is reporting that I have 254 blog pages with duplicate metta titles. Sample below; /blog/2012/06/
On-Page Optimization | | UnderMe
/blog/2012/07/
/blog/2012/08/
/blog/2012/10/
/blog/2012/12/
/blog/2013/01/
/blog/
/blog/?s=boxer
/blog/?s=briefs
/blog/?s=cufflinks
/blog/category/swimwear/
/blog/category/uncategorized/ Can anyone advise what is the best way to address this issue as wordpress seems to assign the site title and tag line set in general settings as the meta title for all blog posts and pages. Thanks0 -
Do I need a Meta description for every page?
HI Guys, We have just developed a new website and I'm looking to add meta descriptions with relevant key words to the pages . As the site has over 80 pages it is quite an undertaking and i was wandering if pages, such as the shopping cart and FAQ's etc, need meta descriptions as well? Thanks in advance : ) Pete
On-Page Optimization | | dawsonski0 -
Meta refresh - nojavascript url
seomox is telling me that I am getting a page that is not being indexed or crawled and since the crawl status code is 200 and there are no robots the meta-refresh url must be the problem. the meta refresh url is different than the on page report card url as it's the nojavascript url which my developer says should be ok. see his comments below. The is redirecting to http://mastermindtoys.com/store/nojavascript.html only in case if the JavaScript is disabled in the client browser. This is the right way to do it, I don’t understand why this might be a problem, otherwise MM has to implement Noscript pages that have a real content. I didn’t get what’s wrong about accessibility. The code 200 means it is accessible, and yes there is nothing to access if JavaScript is disabled on browser. I think there are no modern retail sites that would do any sensible business with the scripting disabled in browsers.The H1 is really present 2 times and second occurrence can be removed, though I highly doubt about importance of this change.Regarding duplicates – what URLs are considered duplicates? Can you please send me examples?I am not aware of canonical URL problem for MM site unless we consider old .asp links as duplicate links of the canonical product pages. I would appreciate if SEOMoz gave us an example what they mean.I suspect that the page is not getting indexed as a result of this or I'm just not getting a good score. Which is it?
On-Page Optimization | | mastermindtoys0 -
Is it ok to use encoded special characters in meta titles?
I've read blog posts stating that encoding special characters in title tags is both ok and not ok. Any definitive answer out there? Do the extra characters from adding encoding count towards the total number of characters that Google displays in SERPs? Or do they just count as one character?
On-Page Optimization | | BostonWright0