Blocked By Meta Robots
-
Hi
I logged in the other day to find that over night I received 8347 notices saying certain pages are being kept out of the search engine indexes by meta-robots.
I have not changed my robots.txt in years and I certainly didn't block Google from visiting those pages.
Is this a fault on Roger Mozbot behalf? Or is there a bot preventing 8000+ pages being indexed?
Is there a way to find out what meta-robot is doing this and where? And how I can get rid of it?
I usually rank between #3 and #5 for the term 'sex toys' on google.com.au, but I now rank #7 to #9 so it would seem some of my pages/content is being blocked.
My website is www.theloveshop.com THIS IS AN ADULT TOYS SITE.
There is no porn videos or anything like that on it, but just in case you don't wish to look at sex toys or are around kids I thought I would mention it.
Blake
-
Hi Nakul
Thank you very much for the suggestions and I will start checking the code on those pages today. Thanks for heads up on the .com.au site as well I will be changing the hosting today to Aus, and thanks to you that site may get pushed up the rankings for a highly competitive term that I have been fighting to get for years
As for following my competitors links, I have been doing that for awhile and it would seem most of them either buy their links or have them on pages that are irrelevant to our niche, one of my competitors has even acquired more links than SEOmoz.com but has no were near the content. In saying that I am by no means a pure white hat SEO, more of a grey hat if such a thing exists, simply because I find it hard to get links to an adult site without it being spammy of some sorts. So I have started writing articles and linking to products mentioned, but trying to make the articles interesting/funny at the same time. Hopefully soon I will start to see some improvement.
Again thank you for your help and I will let you know how I go with those pages blocked by bots.
Blake
-
The meta robots block sounds like it's a block from your CMS or something in your header include file or something. I would check on the pages that show as blocked to see if there's any noindex sort of code in those pages. Other then that, shift in rankings could be based on your competition optimizing for the same set of keywords. I would recommend to spend some time analyzing the backlinks of the competition and see if you can gain some of those kinds of links.
On a side note: It also looks like you own both the .com and the .com.au site. The .com is hosted in AU and the .com.au is hosted in USA. Did you intend to do it the other way around ? I would imagine your rankings for the .com.au get better if it was hosted in Australia instead of USA.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
The Keyword density in a landing page is very low . by adding meta keyword tag can improve keyword density?
The Keyword density in a landing page is very low . by adding meta keyword tag can improve keyword density?
On-Page Optimization | | socialhi50 -
Meta Titles On Site Different From Google Index Page
This is very embarrassing but I hope someone can help. The Meta Titles on my site are not being shown correctly on the Google site index. For example, when I got directly to the page the appropriate page title is shown obviously. However when I go view that page on Google, the title in completely different. Page title: <a class="attribute-value">Web Design, SEO, PPC, Mobile Development In Philadelphia, Bucks| Infinity Digital Agency</a> Google Shows: Our Services - Infinity Digital Agency Here is the result page. I am currently running WordPress and Yoast. Any thoughts would be greatly appreciated. https://www.google.com/search?q=site%3Awww.infinitydigitalagency.com%2Fservices%2F&ie=utf-8&oe=utf-8
On-Page Optimization | | infinitydigitalagency0 -
Different title tags and meta descriptions for desktop and mobile?
Is it possible to use different title tags and meta descriptions for mobile users? For Example: In the SERP for desktop you'll see the desktop title tags and meta descriptions, but in the SERP for mobile you'll the mobile versions of the webpage.
On-Page Optimization | | alex19780 -
Meta Title, Meta Description in Mobile Version
Hello all! We all know that having a responsive theme, with the same URL as the desktop version is what search engines prefer to best index our sites. But how is it when it comes to Meta Titles and Meta Description. Should we use different Metas for mobile visitors? Regards!
On-Page Optimization | | NelsonF0 -
Wordpress Duplicate Meta Title Issue
Hi, Google webmasters tools is reporting that I have 254 blog pages with duplicate metta titles. Sample below; /blog/2012/06/
On-Page Optimization | | UnderMe
/blog/2012/07/
/blog/2012/08/
/blog/2012/10/
/blog/2012/12/
/blog/2013/01/
/blog/
/blog/?s=boxer
/blog/?s=briefs
/blog/?s=cufflinks
/blog/category/swimwear/
/blog/category/uncategorized/ Can anyone advise what is the best way to address this issue as wordpress seems to assign the site title and tag line set in general settings as the meta title for all blog posts and pages. Thanks0 -
Large block of several thousand words on homepage of Ecommerce site - opinions?
I work for a local SEO company who are continually adding content to the home page of clients websites. While i do agree that it is a good idea to add content to the home page and to link to inner pages, the home page of several clients exceeds 6,000 or more words. Every month, an article based on a brand or category (which already has its own page which is optimised with an article) is added to the home page with links to inner pages. I have stressed that it is not a good idea to have so much content on the home page, and even more so that it shouldn't target all the same keywords. I have pointed out that many of the brand and category pages do not rank well for their keywords, whereas in most cases it is the home page which is ranking instead, which i have suggested this is because the home page is too well optimised for those keywords. What are your opinions on this? Are you for or against continually adding content (which already has its own designated page) to the home page of an Ecommerce website? I should also add that this content is within a drop down div box at the footer on some sites and just above the footer on other sites. The category and brand pages also have drop down divs with an article just below the header.
On-Page Optimization | | Kinsel0 -
Information being added before meta description
On the search engine results page I am finding that information has been added before my meta description - this is only a recent flaw and I've no idea how it can be caused. It is happening in the same format as you see dates sometimes before meta descriptions. This is what is actually being shown on the search engine results page. I am getting this text - Product 1 - 15 of 18 – then my meta description after that. The words 'Product 1 - 15 of 18' is in the heading at the top of the page but it is in no way interacting with the meta description. It looks like Google is confusing the words 'Product 1 - 15 of 18' with a date possibly. Any suggestions.
On-Page Optimization | | serp3600 -
The SEOmoz crawler is being blocked by robots.txt need help
SEO moz is showing me that the robot.txt is blocking content on my site
On-Page Optimization | | CGR-Creative0