Quickest way to deindex a large number of pages
-
Our site was recently hacked by spammers posting fake content and bringing down our servers, etc. After a few months, we finally figured out what was going on and fixed the issue. However, it turns out that Google has indexed 26K+ spammy pages and we've lost page rank and search engine rankings as a result.
What is the best and fastest way to get these pages out of Google's index?
-
Given that I'm sure you've removed these pages from your site, there will be no page to which to add a meta-noindex tag.
Disallowing these pages in robots.txt in no way signals to the search engines that they should be removed from the index, just that they should no longer be crawled. Given that they're already indexed, blocking in robots.txt would potentially save some "crawl budget" but wouldn't do anything to remove them from the index.
So submitting them to the URL Removal Tool would be by far the most effective, along with an explanation.
You'll also want to keep a very close watch on your penalty warnings within Webmaster Tools. If you get flagged, you'll want a complete history of the issue and the steps you've taken to address it in order to prepare a reinclusion request.
Lastly, don't forget to submit these same URLs to the Bing Webmaster Tools Block URLs tool. You may not get a massive amount of traffic from Bing, but there's no sense throwing it away, since you've already prepared the URL removal list anyway.
Hope that helps?
Paul
-
Yup. Just wanted to add as well that if these pages are in a particular directory, then you can deindex the entire directory in one command using the URL removal tool.
-
Disallow in robots.txt
Add a noindex meta tag to these pages
Request Google to remove the URLs from their index via WMT URL removal request
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website and landing pages - Proportionate authority
Does website's (homepage) ranking going to influence landing pages ranking or vice-versa? If the homepage is ranking good for a "keyword", will that improve ranking of other landing pages which are optimised for related "keywords" & Vice-versa?
Algorithm Updates | | vtmoz0 -
Product Listing Pages
Hi I had a question regarding product pages and the best way to display the page for SEO. For example, is it best to have a page for - Blue Euro Containers including a table of the capacity options you can buy.. Or, have each product split out so it has it's own product page - 60L Blue Euro Container, etc etc I know a lot of the information will be fairly similar, with the capacity being the one major difference - is this a bad thing? Some of our product tables are too big and the idea was to split them out. Thanks!
Algorithm Updates | | BeckyKey0 -
How is this possible? #2 ranking with NO on-page keywords, no backlinks, no sitemap...
Hi everybody. I have a question ... I'm totally stumped. This question is being asked today (November 16th, 2015) just after Google updated something in their algorithm. Nobody seems to know what they did. and it has something to do with the new "Rank Brain" system they're now using. My niche is Logo Design Software (https://www.thelogocreator.com). I had the keywords "logo creator" on the page roughly 7 times. After Google updated, I lost about 10 spots and as of this writing, I've dropped to #15. So, maybe I over optimized. fine. Noticing that for the keyword "logo creator" ... NONE of the top 14 spots actually have "logo creator" in their page title and NONE of them have more that 2 instances (if any) of the keyword "logo creator" on the actual page. So I removed ALL instances of my keyword "logo creator" from my home page - used the Webmaster's Fetch Tool and moved up a few spots instantly. So what the heck? And the #2 spot for that keyword is www.logomakr.com - they have NO words at all on their pages, no blog, no sitemap and far fewer links than anybody in the top 10. Can anybody reading this shed some light? Marc Marc Sylvester
Algorithm Updates | | Laughingbird
Laughingbird Software0 -
Indexing of Search Pages
I have a question on indexing search pages of an ecommerce or any website. I read Google doesn't recommend this and sites shouldn't allow indexing of their search pages. I recently attended an SEO event (BrightonSEO) and one of the talks was on search pages and how big players like eBay, Amazon do index their search pages. In fact, it is a core part of the pages that are indexed. eBay has to do it, as their product pages are on a time frame and Amazon only allows certain category search pages to be indexed. Reviewing my competitors, they are indexing search pages and this is why they have thousands and millions of web pages indexed. What are your thoughts? I thought search pages were too dynamic (URL strings) and they wouldn't have a unique page title, meta description or rich content to act as a well optimised page. Am I missing a trick here? Cyto
Algorithm Updates | | Bio-RadAbs0 -
Pages fluctuating +/- 70 positions in Google SERPs?
I've got some pages that appear somewhere around #25 in Google. Every now and then, it just goes away from the top 100 results for a few days (even up to a week) and then it comes back. I've got other pages that rank around #8 which falls down to about #75 for a while and then it comes back. But while a page may be gone from the top 100 results in the US, it still ranks at about the same place everywhere else in the world (+/- 10 positions). I've seen this happen in the past but never it happened so often. What gives?!?
Algorithm Updates | | sbrault740 -
How much is Page Rank really worth?
We are in a position to purchase a domain, made of relevant keywords to our company with a current page ranking of 4 for their home page. However in looking at their analytics and other information they do not do well on significant keywords and have very low site traffic. In fact they do very, very poorly. With their high page ranking would it be relatively easy to conduct a successful SEO campaign on the domain if we were to take it over as our own and attempt to climb in the SERP's? I know Page Rank doesn't mean everything when it comes to your ranking, but 4 is relatively high in our field, so I don't really understand why they do so poorly when it comes to their actual rankings on key words.
Algorithm Updates | | absoauto0 -
Google has indexed a lot of test pages/junk from the development days.
With hind site I understand that this could have been avoided if robots.txt was configured properly. My website is www.clearvisas.com, and is indexed with both the www subdomain and with out. When I run site:clearvisas.com in Google I get 1,330 - All junk from the development days. But when I run site:www.clearvisas.com in Google I get 66 - these results all post development and more in line with what I wanted to be indexed. Will 1,330 junk pages hurt my seo? Is it possible to de-index them and should I? If the answer is yes to any of the questions how should I proceed? Kind regards, Fuad
Algorithm Updates | | Fuad_YK0 -
Has anyone notice their website page rankings in Bing and Yahoo slipping lately?
My websites for the most part rank higher in Yahoo and Bing than in Google. Now, ranking within those search engines (Yahoo/Bing) are settling in to where my websites rank in Google. Any thoughts? Anyone else experiencing this? Thank you for your time!
Algorithm Updates | | APICDA0