100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Webmaster Tools Parameters
We have several large ecommerce websites, and we've added some tracking parameters to GWT for google to ignore. All pages are correctly canonicaled. Google has been ignoring the parameters and the canonicals, and still ranks many parametered pages for us. Has anyone run into this?
Intermediate & Advanced SEO | | AMHC0 -
Is it safe to link my websites together?
Hi Everyone, I have 10 websites which are all of good standing and related. My visitors would benefit of knowing about the other websites but I don't want to trigger a google penalty by linking them all together. Ideally I'd also like to pass on importance through the links as well. How would you proceed in this situation? Advice would be greatly appreciated, Peter.
Intermediate & Advanced SEO | | RoyalBlueCoffee0 -
Penguin Apply To Internal Linking?
Is Penguin focused primarily on backlinks or does it also assess internal linking/anchor text? We've lost about 3,000 visitors a month since the rolling updates were implemented. I'm always careful not to over-react to algo updates but enough time has passed that I think the dust has settled. I try to stay white in all I do but I think if I've over-done anything its the internal linking related products/categories with exact match. My backlink profile also has an over-abundance of affiliate links but that's kind of out of my hands isn't it?
Intermediate & Advanced SEO | | AWCthreads0 -
Technical Question on Image Links - Part of Addressing High Number of Outbound Links
Hi - I've read through the forum, and have been reading online for hours, and can't quite find an answer to what I'm searching for. Hopefully someone can chime in with some information. 🙂 For some background - I am looking closely at four websites, trying to bring them up to speed with current guidelines, and recoup some lost traffic and revenue. One of the things we are zeroing in on is the high amount of outbound links in general, as well as inter-site linking, and a nearly total lack of rel=nofollow on any links. Our current CMS doesn't allow an editor to add them, and it will require programming changes to modify any past links, which means I'm trying to ask for the right things, once, in order to streamline the process. One thing that is nagging at me is that the way we link to our images could be getting misconstrued by a more sensitive Penguin algorithm. Our article images are all hosted on one separate domain. This was done for website performance reasons. My concern is that we don't just embed the image via , which would make this concern moot. We also have an href tag on each to a 'larger view' of the image that precedes the img src in the code, for example - We are still running the numbers, but as some articles have several images, and we currently have about 85,000 articles on those four sites... well, that's a lot of href links to another domain. I'm suggesting that one of the steps we take is to rel=nofollow the image hrefs. Our image traffic from Google search, or any image search for that matter, is negligible. On one site it represented just .008% of our visits in July. I'm getting a little pushback on that idea as having a separate image server is standard for many websites, so I thought I'd seek additional information and opinions. Thanks!
Intermediate & Advanced SEO | | MediaCF0 -
301 Redirect and Webmaster Central
I've been working on removing canonical issues. My host is Apache. Is this the correct code for my htaccess? RewriteEngine On
Intermediate & Advanced SEO | | spkcp111
RewriteCond %{HTTP_HOST} ^luckygemstones.com$ [NC]
RewriteRule ^(.*)$ http://www.luckygemstones.com/$1 [R=301,L] SECOND!!! I have two websites under Google's Webmaster Central; http://luckygemstones.com which gets NO 404 soft errors... AND http://www.luckygemstones.com which has 247 soft 404 errors... I think I should DELETE the http://luckygemstones.com site from Webmaster Central--the 301 redirect handles the"www" thing. Is this correct? I hate to hose things (even worse?) Help! Kathleen0 -
Help Identifying Unnatural Links
http://bit.ly/XT8yYYHi,Any help with the below will be most appreciated.We received an unnatural links warning in Webmaster Tools and noticed a large drop in our rankings. We downloaded and carried out a full link audit (3639 links) and logged in an excel spreadsheet with the following status: OK, Have Contacted, Can't Contact, Not SureWe have had some success but the majority of the ones we identified are not contactable.We use the dis-avow tool to tell Google of these. We then submitted a reconsideration request where we explained to Google our efforts and that we can supply them with our audit if necessary by email as you can't upload any evidence.A few days later we received a response suggesting that we still have unnatural links. We are a little stuck as we don't know what they can be:1. Is Google actually looking at our dis-avowed links before making this judgement?2. We have missed something that Google is considering bad but we can't see in our audit?Again we need a little help as we are trying to sort this out but can't see what we are falling down on.I can provide our spreadsheet if necessary.Many ThanksLee
Intermediate & Advanced SEO | | LeeFella0 -
Real impact of canonical links?
I am responsible for 2 e-commerce websites. SEO Moz and Google Web Master tools both inform me regularly that on both sites there are many instances of duplicate titles, headings, decriptions and page content. Obviously from an SEO point of view I am more than a little concerned about this! Out product pages struggle to perform strongly despite the fact that our website is of a decent quality and we are leaders in our field. Our competitors rank above us when they add a product page, whereas we normal flit in between 8-10 or on the 2nd SERP. I know it is hard without viewing the site, but is duplicate content likely to be a strong, leading factor in this? I think it is, but want to put together a business case to spend the cash to sort it out....just need someone confirmation that this is worth sorting as a priority. Here are 2 examples of what I mean: 1) Category pages www.exampledomain.co.uk/category1.aspx We have filters on our category page (so the customer can sort products based on their price, colour, size etc.). When filters are used a new URL is generared. www.exampledomain.co.uk/category1.aspx?prices=0||10 www.exampledomain.co.uk/category1.aspx?prices=10||20 The content, titles, description is the same although the links are different. Do I need to set up a canonical tag on the page that reads: 2) Product pages Product pages on the websites have different URLs depending on how to arrive on them. You get 1 URL if you navigated to the page via the website navigation, but you get another different URL if you used the website search functionality to find the page. Example: Search link: www.exampledomain.co.uk/category1/Product1.aspx Navigation link: www.exampledomain.co.uk/12345/category1/Product1.aspx Again, do I need to set up a canonical tag for 1 of these link types so that the link benefit is not shared over 2 pages? Any feedback would be welcome! At the moment the ability to add canonical tags is locked down by our CMS (I know, rubbish!)...so website development would be needed - hence the need for a business case!
Intermediate & Advanced SEO | | DHS_SH0 -
Too many links!
Hi, I'm running a wordpress blog (modhop.com) and am getting the "too many links" on almost all of my pages. It appears that in addition to basic site navigation I have plug-ins that create invisible links that are counted in the crawl...at least that's my guess. Is there a good way to control this in wordpress? A nofollow in the .htaccess? A plug-in that does this? (I'm sort of at novice-plus level here so the simplest solution is ideal.) Thanks! Jake modhop.com
Intermediate & Advanced SEO | | modhop0