Very well established blog, new posts now being indexed very late
-
I have an established blog.We update it on daily basis. In the past, when I would publish a new post, it would get indexed within a minute or so.
But since a month or so, its taking hours. Sometimes like 10-12 hours for new posts to get indexed. Only thing I have changed is robots.txt.
This is the current robots file.
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /wp-login.php Disallow: /*wp-login.php* Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /author Disallow: /category Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /login/ Disallow: /wget/ Disallow: /httpd/ Disallow: /*.php$ Disallow: /*?* Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.gz$ Disallow: /*.wmv$ Disallow: /*.cgi$ Disallow: /*.xhtml$ Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads User-agent: TechnoratiBot/8.1 Disallow: # ia_archiver User-agent: ia_archiver Disallow: / # disable duggmirror User-agent: duggmirror Disallow: / # allow google image bot to search all images User-agent: Googlebot-Image Disallow: /wp-includes/ Allow: /* # allow adsense bot on entire site User-agent: Mediapartners-Google* Disallow: Allow: /* Sitemap: http://www.domainname.com/sitemap.xml.gz
Site has tons of backlinks. Just wondering if something is wrong with the robots file or if it could be something else.
-
The robots.txt file is designed to completely block content. Normally, if your robots.txt file was a factor then your content would not appear in SERPs at all.
It is possible for content to appear in SERPs even though it is blocked by robots.txt if it is linked from other sources. Since this is new content, it is less likely that is the case unless you are immediately sharing links and Google is seeing those links within the time frame you shared.
The first place I would look is your sitemap or whatever tool is used to inform Google that you have new content. When you publish a new blog article, your software should ping Google and inform them there is new content. That is where any investigation should begin. Next step is to check server logs to see how long it takes Google to respond to the alert. If it takes them 12 hours, then there is nothing further you can do about it.
I would be interested in a lot more detail. How many articles how you confirmed as being affected by this issue. Exactly how did you confirm the issue?
As a side note, your robots.txt file is bloated and doesn't adhere to any standards I have seen. How exactly was it created? Did someone go in and make manual modifications to the file?
-
Are you using Feedburner? Has the feed publishing service gotten out of sync? You can re-sync it under the Troubleshootize section.
-
Yes, its a wordpress site and I always had the all in one SEO plugin enabled.
-
Do you use a word press platform? If so do you use a SEO plug in. Different plug ins can effect the index time.
-
Do you use a word press platform? If so do you use a SEO plug in. Different plug ins can effect the index time.
-
Could you possibly reverse the changes of the robots.txt to a previous "working" version where your site was getting indexed quicker?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do you Index your Image Repository?
On our backend system, when an image is uploaded it is saved to a repository. For example: If you upload a picture of a shark it will go to - oursite.com/uploads as shark.png When you use a picture of this shark on a blog post it will show the source as oursite.com/uploads/shark.png This repository (/uploads) is currently being indexed. Is it a good idea to index our repository? Will Google not be able to see the images if it can't crawl the repository link (we're in the process of adding alt text to all of our images ). Thanks
Technical SEO | | SteveDBSEO0 -
I have a Category and Tag In My Blogs
I have use category and Tags in my blogs. Now i have an problem with blog URL and Tags URL. My blog URLs is also show in Tags page and both the content is same. For Example: My Blog URL is: https://www.example.com/advice-how-to-do-batting And Tag Page URL is : https://www.example.com/advice-batting in that - https://www.example.com/advice-how-to-do-batting The URLs contain same content. No should i write two different meta title and description for above two URLs pages. As there might more blog added under Tags pages with different topics and title. Request on Thought Please.
Technical SEO | | ProcessSEO0 -
Staging & Development areas should be not indexable (i.e. no followed/no index in meta robots etc)
Hi I take it if theres a staging or development area on a subdomain for a site, who's content is hence usually duplicate then this should not be indexable i.e. (no-indexed & nofollowed in metarobots) ? In order to prevent dupe content probs as well as non project related people seeing work in progress or finding accidentally in search engine listings ? Also if theres no such info in meta robots is there any other way it may have been made non-indexable, or at least dupe content prob removed by canonicalising the page to the equivalent page on the live site ? In the case in question i am finding it listed in serps when i search for the staging/dev area url, so i presume this needs urgent attention ? Cheers Dan
Technical SEO | | Dan-Lawrence0 -
What do we do now?
OK where do we start? Lets go back to June 2011So in June 2011 we left an SEO company that was looking after our account, after a year of being with them and not moving 1 place in the SERPS we decided to cut ties with them and move to a new SEO company... Over the first 6 months things seemed to be going well, for our main keyword "car warranty" we started to climb up the rankings from #6 - #4 - #2, we also moved up for other keywords. One thing that I did notice when we were with the new SEO Company not much on-site optimisation was completed.Move forward to January 2012So over the Christmas period the usual happened, our rankings stayed the same put traffic dropped which is obviously normal for this specific time of year... But on the 7th of Jan 2012 our rankings dropped from #2 - #10, we contacted the SEO company in question and they reported back that a server that had 20links pointing back to our server had crashed on Christmas Day and the links had been de-indexed from Google, they said give it 2 - 3 weeks and the links will be re-indexed and we should pop back up on google.3 Weeks later we were still in the exactly the same position, the good thing was because we run a very good Adwords campaign the traffic to our site didn't drop.The SEO company then came back and said it looks like we have been placed in a 60day filter by Google and once the 60day filter had been reached we should pop back up - 65days came and went and we were still in the exact same position. After us waiting around for months to see if the rankings improved we decided to leave the SEO company and move all our SEO work back in-house.Move forward to February 2012Once we had full control over the account again we made some changes to the on-site optimisation of the site, we improved page titles, descriptions, tags and also our content was re-written - we then waited for Google to pickup the changes and re-index our site. YesterdaySo yesterday we checked our rankings - some of our longer tail keywords had improved but our main big traffic keywords had dropped even further - we had gone from #10 to #16, with the update to the algorithm yesterday targeting spammy websites we feared that we had been hit by the new update. We then cleaned up some links that looked spammy and asked for a reconsideration request. Now looking deeper into our backlinks we still have some spammy non-relevant links as well as a few big sitewide links - 1 sitewide link provides 732,667 links to our homepage and our total links indexed by Google is only 759,144.What do you think we should do...Wait for Google to come back to us after the reconsideration request?Remove more and backlinks?Build more high value links?If anyone can provide me with more information it would be great.Thanks,
Technical SEO | | ScottBaxterWW
Scott0 -
New project old domain should I 301 redirect while new sites built
I just took on a larger scale e-commerce project and came across a tricky road block that I need some advise on. Ok I'm building the site from scratch and due to it's complexity it may take 3-4 months before I have it designed and coded. The client has a domain name that has some decent page/domain authority and I would hate to loose that while the sites being built. Currently I have nothing to display as his previous site got hacked and it was deleted by the previous web admin. Being that a blog has already been approved as part of the project I already installed wordpress to keep the domain fresh however here's the issue, I installed wordpress in a folder called blog and debating if I should 301 redirect or 302 redirect his index here? The blog will always reside in the blog folder even after launch. Will performing a 301 redirect pull all the juice away from my index page? I'm assuming yes. IF so what would occur once the project is complete and I make the ecommerce site live in the index page? Thanks in Advance! Mike
Technical SEO | | MikeDelaCruz770 -
Best Practice to Remove a Blog
Note: Re-posting since I accidentally marked as answered Hi, I have a blog that has thousands of URL, the blog is a part of my site. I would like to obsolete the blog, I think the best choices are 1. 404 Them: Problem is a large number of 404's. I know this is Ok, but makes me hesitant. 2. meta tag no follow no index. This would be great, but the question is they are already indexed. Thoughts? Thanks PS A 301 redirect to the main page would be flagged as a soft 404
Technical SEO | | Bucky0 -
What to do with Deleted Posts?
Hi Guys, To give way to the implementation of Google Panda, I have deleted some of my wordpress blog posts with low quality content. Now, I am seeing some errors in my Webmasters Tools. What needs to be done so that these errors will be fixed? Thanks....
Technical SEO | | Trigun0 -
Does Google index XML files?
Does Google or other search engines include XML files in their index? More specifically, I am wondering how Google knows the difference between an xml filetype and an RSS feed.
Technical SEO | | nicole.healthline0