How can I get unimportant pages out of Google?
-
Hi Guys,
I have a (newbie) question, untill recently I didn't had my robot.txt written properly so Google indexed around 1900 pages of my site, but only 380 pages are real pages, the rest are all /tag/ or /comment/ pages from my blog. I now have setup the sitemap and the robot.txt properly but how can I get the other pages out of Google? Is there a trick or will it just take a little time for Google to take out the pages?
Thanks!
Ramon
-
If you want to remove an entire directory, you can exclude that directory in robots.txt, then go to Google Webmaster Tools and request a URL removal. You'll have an option to remove an entire directory there.
-
No, sorry. What I said is, if you mark the folder as disalow in robots.txt, it will not remove the pages are already indexed.
But the meta tag, when the spiders go again on the page and see that the pages are with the noindex tag will remove it.
Since you can not already include the directory on the robots.txt. Before removing the SE pages.
First you put the noindex tag on all pages you want to remove. After they are removed, it takes a week for a month. After you add the folders in robots.txt to your site who do not want to index.
After that, you dont need to worry about the tags.
I say this because when you add in the robots.txt first, the SE does not read the page anymore, so they would not read the meta noindex tag. Therefore you must first remove the pages with noindex tag and then add in robot.txt
Hope this has helped.
João Vargas
-
No, sorry. What I said is, if you mark the folder as disalow in robots.txt, it will not remove the pages are already indexed.
But the meta tag, when the spiders go again on the page and see that the pages are with the noindex tag will remove it.
Since you can not already include the directory on the robots.txt. Before removing the SE pages.
First you put the noindex tag on all pages you want to remove. After they are removed, it takes a week for a month. After you add the folders in robots.txt to your site who do not want to index.
After that, you dont need to worry about the tags.
I say this because when you add in the robots.txt first, the SE does not read the page anymore, so they would not read the meta noindex tag. Therefore you must first remove the pages with noindex tag and then add in robot.txt
Hope this has helped.
João Vargas
-
Thanks Vargas, If I choose for noindex, I should remove it from the robot.txt right?
I understood that if you have a noindex tag on the page and as well a dissallow in the robot.txt the SE will index it, is that true?
-
For you remove the pages you want, need to put a tag:
<meta< span="">name="robots" content="noindex">If you want internal links and external relevance to pass on these pages, you put:
<meta< span="">name="robots" content="noindex, follow">If you do the lock on robot.txt: only need to include the tag in the current urls, new search engines will index no.
In my opinion, I do not like using the google url remover. Because if someday you want to index these folders, will not, at least it has happened to me.
The noindex tag works very well to remove objectionable content, within 1 month or so now will be removed.</meta<></meta<>
-
Yes. It's only a secondary level aid, and not guaranteed, yet it could help speed up the process of devaluing those pages in Google's internal system. If the system sees those, and cross-references to the robots.txt file it could help.
-
Thanks guys for your answers....
Alan, do you mean that I place the tag below at all the pages that I want out of Google? -
I agree with Alan's reply. Try canonical 1st. If you don't see any change, remove the URLs in GWT.
-
There's no bulk page request form so you'd need to submit every URL one at a time, and even then it's not a guaranteed way. You could consider gettting a canonical tag on those specific pages that provides a different URL from your blog, such as an appropriate category page, or the blog home page. That could help speed things up, but canonical tags themselves are only "hints" to Google.
Ultimately it's a time and patience thing.
-
It will take time, but you can help it along by using the url removal tool in Google Webmaster Tools. https://www.google.com/webmasters/tools/removals
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why google removed my landing pages from index?
I made new website meko.lv. I put many work to it, to make page SEO friendly, sprites, reduced requests added SSL, got google page speed insights score 100/100, but in 2. october all pages in google webmasters disappeared from index. Could you please look at website and say whats wrong with it? They are all search results present in google but for how long. it is so annoying, you put so many work but in result get high spam score. It is obvious that new pages can not get good links in one month https://meko.lv/ google webmasters google page speed score: https://developers.google.com/speed/pagespeed/insights/?url=http%3A%2F%2Fmeko.lv%2F&tab=mobile q1LDHTn
Technical SEO | | Mekounko0 -
Beating big brands for rankings on Google page 1 post Panda & Penguin
Hi all, so having followed lots of SeoMoz guidelines that we have read here and standard SEO ideas we seem to no longer be able to rank for our core keywords.. and certainly not rank in front of the big brands. We're a small eCommerce company and have historically ranked Google positions 1-4 for many of our keywords (a year or two ago)... but now no where near this any more. We always write unique content for our products of usually around 300-400 words per product we include our keywords in Title, meta description and H1 tags. We include buyers guides and set up articles on the site and generally have a reasonable amount of good quality and always uniquely written content Recently we have concentrated to ensure that page load speed is above average and Google Web Master Tools page speed gives us around 80-90 out of 100 We carry out linking and have always done... in the most recent past this has been weighted towards 'content for links' to gain purely incoming links (although in the early days from 2005 we did swap links with other web masters as well as write and publish on article sites etc). product category pages have an intro piece of text that includes the key phrases for that page and is placed as close to the body tag as possible. From what I understand if you are hit by Panda or Penguin the drop off is invariably over night, but we have not seen this... more of a gradual decline over the last year or two (although there was a bit of a downward blip on Panda update 20). Now we're lucky to be on page 2 for what were our main keywords / phrases such as "portable DVD players" or "portable DVD player"... in front of us in every position is a big national brand.. and certainly on page 1 it is purely only a big brand in every postion. They don't have great info from what we can see for these keywords and certainly don't give as much info as we do. For the phrase "portable DVD player" our portable DVD accessories page ranks better than our actual portable DVD player category page... which we also can't understand? This is our portable DVD category page: http://www.3wisemonkeys.co.uk/portable-dvd-players-car Currently we're starting to produce 2 minute product demo videos for as many of our product detail pages as we can and we plan to host these on something such as Vimeo so that content will be unique to our site (rather than YouTube) in order to give us a different format of unique content on many of our product detail pages to improve rankings (and conversion rates as the same time ideally). So ... I am hoping that some one out there can point us in the right direction and shed some light on our declining positions. Are we doing or have done something wrong... or is it in these post Panda / Penguin days extremely difficult for a small business to beat the big brands as Google believes these are what every one wants to see when shopping? Thanks for any comments and / or help.
Technical SEO | | jasef0 -
Can Page Content & Description Have Same Content?
I'm studying my crawl report and there are several warnings regarding missing meta descriptions. My website is built in WordPress and part of the site is a blog. Several of these missing description warnings are regarding blog posts and I was wondering if I am able to copy the first few lines of content of each of the posts to put in the meta description, or would that be considered duplicate content? Also, there are a few warnings that relate to blog index pages, e.g. http://www.iainmoran.com/2013/02/ - I don't know if I can even add a description of these as I think they are dynamically created? While on the subject of duplicate content, if I had a sidebar with information on several of the pages (same info) while the content would be coming from a WP Widget, would this still be considered duplicate content and would Google penalise me for it? Would really appreciate some thoughts on this,please. Thanks, Iain.
Technical SEO | | iainmoran0 -
Does google like Category pages or pages with lots of Products on them?
We are having an issue with getting Google to rank the page we want. To have this page http://www.jakewilson.com/c/52/-/346/Cruiser-Motorcycle-Tires rank for the key word Cruiser Motorcycle Tires; however, this page http://www.jakewilson.com/t/52/-/343/752/Cruiser-Motorcycle-Tires is ranking instead and it has less links and page authority according to site explorer and it is farther down in the hierarchy. I am wondering if google just likes pages that have actual products on them instead of a page leading to the page with all the products. Thoughts?
Technical SEO | | DoRM0 -
Why Can't I Get on Google?
I've employed many of the suggestions of SEOMoz and getting a Grade "A" on a particular keyword. I'm now #4 on Yahoo and Bing. However, my site hasn't cracked the top 50 in Google. Why? I see a similar pattern with other keywords, many on yahoo and bing but only a few of my subpages get #45-48 on Google. Any ideas? http://www.gospelebooks.net
Technical SEO | | mrjgardiner0 -
Will Google Continue to Index the Page with NoIndex Tag Upon Google +1 Button Impression or Click?
The FAQs for Google +1 button suggests as follows: "+1 is a public action, so you should add the button only to public, crawlable pages on your site. Once you add the button, Google may crawl or recrawl the page, and store the page title and other content, in response to a +1 button impression or click." If my page has NoIndex tag, while at the same time inserted with Google +1 button on the page, will Google recognise the NoIndex Tag on the page (and will not index the page) despite the +1 button's impression or clicks send signals to Google spiders?
Technical SEO | | globalsources.com0 -
Google indexing directory folder listing page
Google somehow managed to find several of our images index folders and decided to include them into their index. Example: websitesite.com/category/images/ is what you'll see when doing a site:website.com search. So, I have two-part question: 1) Does this hurt our site's ability to rank in any way?
Technical SEO | | invision
Because all Google sees is just a directory listing page with a bunch of links to images in the folder. 2) If there could be any negative effect, what is the best way to get these folders out of Google's index?
I could block via robots.txt, but I'm afraid it will also block all the images in that folder from being indexed in Google image search. I could also turn off directory listing in cpanel / htaccess, but then that gives is a 403 forbidden. Will this hurt the site in anyway and would it prevent Google from indexing the images in the directory? Thanks,
Tony0 -
Can someone break down 'page level link metrics' for me?
Sorry for the, again, basic question - can someone define page level link metrics for me?
Technical SEO | | Benj250