Disallow: /search/ in robots but soft 404s are still showing in GWT and Google search?
-
Hi guys, I've already added the following syntax in robots.txt to prevent search engines in crawling dynamic pages produce by my website's search feature: Disallow: /search/. But soft 404s are still showing in Google Webmaster Tools. Do I need to wait(it's been almost a week since I've added the following syntax in my robots.txt)? Thanks, JC
-
You could also look at using the meta robots = noindex tag on /search/ pages, rather than just blocking it in robots.txt, as this will remove existing URLs from the index.
-
Glad to help
-
Thanks a lot Dan!
-
That is a good recommendation but ultimately search engines will make a final decision on crawl frequency. Take a look at your 'Crawl Stats' on GWTs and this will give you an idea of how often your site is crawled.
-
Is the time issue related in crawl frequency of the URLs in my sitemap?
Thanks Dan, appreciate it.
-
You will probably need to wait a little longer - it depends how often your site usually gets crawled and indexed.
However, robots.txt does not always stop search engines from indexing your pages. It will stop them crawling a page on your site but it tells them that they can still index that page. If they find links from external sites then the URL may still appear in the SERP.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Console Showing 404 errors for product pages not in sitemap?
We have some products with url changes over the past several months. Google is showing these as having 404 errors even though they are not in sitemap (sitemap shows the correct NEW url). Is this expected? Will these errors eventually go away/stop being monitored by Google?
Technical SEO | | woshea0 -
Abnormally high internal link reported in Google Search Console not matching Moz reports
If I'm looking at our internal link count and structure on Google Search Console, some pages are listed as having over a thousand internal links within our site. I've read that having too many internal links on a page devalues that page's PageRank, because the value is divided amongst the pages it links out to. Likewise, I've heard having too many internal links is just bad in general for SEO. Is that true? The problem I'm facing is determining how Google is "discovering" these internal links. If I'm just looking at one single page reported with, say, 1,350 links and I'm just looking at the code, it may only have 80 or 90 actual links. Moz will confirm this, as well. So why would Google Search Console report different? Should I be concerned about this?
Technical SEO | | Closetstogo0 -
GWT Soft 404 count is climbing. Important to fix?
In GWT I am seeing my mobile site's soft 404 count slowly rise from 5 two weeks ago to over 100 as of today. If I do nothing I expect it will continue to rise into the thousands. This is due to there being followed links on external sites to thousands of discontinued products we used to offer. The landing page for these links simply says the product is no longer available and gives links to related areas of our site. I know I can address this by returning a 404 for these pages, but doing so will cause these pages to be de-indexed. Since these pages still have utility in redirecting people to related, available products, I want these pages to stay in the index and so I don't want to return a 404. Another way of addressing this is to add more useful content to these pages so that Google no longer classifies them as soft 404. I have images and written content for these pages that I'm not showing right now, but I could show if necessary. But before investing any time in addressing these soft 404s, does anyone know the real consequences of not addressing them? Right now I'm getting 275k pages indexed and historically crawl budget has not been an issue on my site, nor have I seen any anomalous crawl activity since the climb in soft 404s began. Unchecked, the soft 404s could climb to 20,000ish. I'm wondering if I should start expecting effects on the crawl, and also if domain authority takes a hit when there are that many soft 404s being reported. Any information is appreciated.
Technical SEO | | merch_zzounds0 -
Google appending keyword to local search result(s)?
I noticed an interesting change today in how one of my clients appears in the SERPs. Google seems to be appending a keyword to his listing title. Client website: www.mycalgarydentist.com Keyword: Calgary dentist Rank: #2 or #1 lately Title tag: Calgary Dentist | Ambiance Dental Google+ Local listing title: Ambiance Dental Link title in SERP: Ambiance Dental: Calgary Dentist That last point is what's interesting, and new. As of a couple weeks ago (before I went on holidays) his link would simply show "Ambiance Dental", which makes sense because that's the title of his Google+ Local listing. Given the above information, I can't see why his link in Google's SERP is "Ambiance Dental: Calgary Dentist" when doing a search for that keyword. When I do a search for "Calgary dentists" or other similar searches, he simply shows as "Ambiance Dental", not "Ambiance Dental: Calgary Dentists" To test yourself, use the Google AdWords Preview Tool (https://adwords.google.com/d/AdPreview/), change locality to "Calgary, AB, Canada" and search. I suspect this doesn't mean he's violating Google's guidelines for business listings (i.e. businesses aren't supposed to add keywords to their business title). I'm certainly curious why this is happening though. Can anyone provide any insight? Has anyone seen anything similar? calgary-dentist-search.png
Technical SEO | | Kenoshi0 -
Blocked by meta-robots but there is no robots file
OK, I'm a little frustred here. I've waited a week for the next weekly index to take place after changing the privacy setting in a wordpress website so Google can index, but I still got the same problem. Blocked by meta-robots, no index, no follow. But I do not see a robot file anywhere and the privacy setting in this Wordpress site is set to allow search engines to index this site. Website is www.marketalert.ca What am I missing here? Why can't I index the rest of the website and is there a faster way to test this rather than wait another week just to find out it didn't work again?
Technical SEO | | Twinbytes0 -
Business name not showing on Google Maps Satellite View
Hi Everyone, When you go into Google Maps and look at the satellite view, you'll notice business names on some, but not all, businesses--like on the roof of the building. Where do I need to go to get this to work for my business? My Google Places pages are setup properly but as far as I can tell there is no setting for this. Is it the Service Area setting? Currently I have it set to "Yes, this business serves customers at their locations." Perhaps I need to set this to "No." Many thanks, Robert
Technical SEO | | AC_Pro1 -
Every time google caches our site it shows no website.
Our site <cite>www.skaino.co.uk/</cite> seems to be having real issues with being picked up with Google. The site has been around for a long time but no longer even ranks on google if you search for the word 'Skaino'. This is odd as its hardly a competitive keyword. If I do a site:www.skaino.co.uk then it shows all the pages proving the site has been indexed. But if I do cache:www.skaino.co.uk it shows a blank cache. I'm starting to worry that Google isn't able to crawl our site properly. If it helps to clarify we have a flash site with a HTML site running underneath for those who cant view flash. Im wandering if I've missed something glaringly obvious. Is it normal to have a blank google cache? Thanks AJ
Technical SEO | | handygammon0 -
Do search engines still index/crawl private content?
If you have a membership site, which requires a payment to access specific content/images/videos, do search engines still use that content as a ranking/domain authority factor? Is it worth optimizing these "private" pages for SEO?
Technical SEO | | christinarule1