Get a list of robots.txt blocked URL and tell Google to crawl and index it.
-
Some of my key pages got blocked by robots.txt file and I have made required changes in robots.txt file but how can I get the blocked URL's list.
My webmaster page Health>blocked URL's shows only number not the blocked URL's.My first question is from where can I fetch these blocked URL's and how can I get them back in searches,
One other interesting point I see is that blocked pages are still showing up in searches.Title is appearing fine but Description shows blocked by robots.txt file.
I need urgent recommendation as I do not want to see drop in my traffic any more.
-
"changing the lastmod of those pages to today".
How can I make these changes?
Right now the news is that Resubmitted the Sitemap and no warnings this time.
-
I imagine that since you've got a robots txt error you'll probably ended closing a whole directory to bots which you wanted to be indexed. You can easily spot the directory and resubmit a sitemap to google changing the lastmod of those pages to today and the priority to 1 but only of those pages.
If you still receive warnings it may be due to errors in your sitemap. You're probably including some directory you don't want. You can try it in GWT putting in the box at the bottom the url you want to maintain in the index and then trying to see if some urls are being blocked by your robots.
If you want you can post here your robots and the URIs you want to be indexed without knowing the domain so that won't be public. Hope this may help you
-
Ok Resubmitted it.but even with updated file it gives a lot of errors.I think it takes some time.20,016 warnings
I have not added no index attribute in my header region.It was all messy stuff with robots.txt file.It means that with site showing up in SERP the rank will probably be the same or it was deranked?
-
Go into GWMT and resubmit sitemap.xml files (with the URLs you want indexed) for recrawling and Google will digest the sitemaps again, instead of waiting for Googlebot to come around on their own, you are requesting it to come around, also include those new sitemap files in your robots.txt file.
-
In Google Webmaster Tools, go to Health -> Fetch As Google. Then add the previously blocked URL and click Fetch. Once you've done that, refresh the page and click "Submit to index". That should get Google indexing those pages again.
Getting external links to your pages also helps get pages crawled & indexed, so it may be worth submitting your pages to social bookmarking sites, or get other types of backlinks to your previously blocked pages if possible.
-
Since you fixed your robots.txt file you should be good to. It will probably take a few days for Google to recrawl your site and update the index with the URLs they are now allow to crawl.
Blocked URLs can still show up in SERPs if you haven't defined the no-index attribute in your section.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Analytics SEO Queries Not Showing
Hi All, This might be a silly question, but for all the properties I monitor in Google Analytics, I'm now showing no data for SEO Queries under Acquisition for the past 6 days. Normally I would expect a few day delay in queries, but nothing for 6 days is somewhat peculiar especially as it was functioning fine prior to November 12th. Does anyone have insight into what might be going on? Thanks! URaNMa3
Reporting & Analytics | | amichaels0 -
Google Analytics - Average Position
Hi Just trying to get some clarity on Google Analytics Average Positions in "Aquistions/Search Engine Optimisation". For a very competitive keyword Google Analytics is saying i am on average position of 6. Is this Page 6? I am assuming position six would be 1.6?
Reporting & Analytics | | Cocoonfxmedia0 -
How many users completely block Google Analytics cookies ?
Hello everyone! In your experience, how many of your visitors' browsers completely block cookies including those of Google Analytics ?
Reporting & Analytics | | Masoko-T0 -
Using Regex for Goals in Google Analytics
I’ve got a website with 20 different forms; and I’d like to track all 20 form completions with one Goal in Google Analytics. The completion-page URLs follow the pattern below (the website uses Drupal CMS) Form One: www.acme.com/node/2990/done?sid=651 Form Two: www.acme.com /node/2991/done?sid=785 Form Three: www.acme.com /node/2992/done?sid=1021 Form Four: www.acme.com /node/2993/done?sid=459 I believe that there is a way I can use regex so Google Analytics will track if any of these completion URLs appears. Looking for guidance how to format the regex statement.
Reporting & Analytics | | TopFloor0 -
My own brand searches stopped appearing in google
Hi everyone, I've never asked a question here before, so go easy on me 🙂 I've noticed recently that my site is not showing up for my own brand searches anymore, and that my organic google results are way down. For example, it used to be if you would search for "grouvee kings quest 6" you would either see http://www.grouvee.com/news/kings-quest-6-retro-rerun-playthrough-part-1/63/ or http://www.grouvee.com/games/kings-quest-vi-heir-today-gone-tomorrow/69/ show up first in google. Now they're nowhere to be found. Here's another example. If you would search for "grouvee mass effect 3" you'd either see http://www.grouvee.com/games/mass-effect-3/75/ or http://www.grouvee.com/news/mass-effect-3-lets-take-the-earth-back/136/ show up first in google. Once again, nowhere to be found. I could go on with several other examples, but I don't really know what I've done. I 301 redirected 2 pages a few weeks ago because I added a /reviews/ structure to my site and needed to move a couple of pages that used to be under /news/ over to the new section, but those shouldn't have had anything to do with the pages I referenced above. I haven't done any link building really. I submit some of my articles occasionally to places like reddit or n4g, but that's about it. Anyone have any tips or things I should be looking at?
Reporting & Analytics | | petecorsaro0 -
Analytics URL Tagging
For some reason I can't get Google Analytics to pick up my URL tags, am I doing something wrong? http://www.example.com/?utm_source=carscom&utm_campaign=3rdparty&utm_medium=referral
Reporting & Analytics | | kylesuss0 -
Weekly crawl or monthly crawl?
Dear SEOmoz, Thanks for all the stuff in this pro account. I tweak my websites everyday and do a lot of linkbuilding. But I noticed the statistics don't change weekly. I've been using SEOmoz tools now for maybe two months and there has been one update in stats since. Am I too impatient or is it a monthly crawl? ^^ Thanks again! PS: Mostly talking about the opensiteexplorer.org statistics
Reporting & Analytics | | erikwigman0 -
Tagging URLs Linkbuilding and anchor links
Hi, I am going to publish a press release on a number of different websites. First and foremost, I want to build anchor links back to website for specific keywords. Secondly I want to measure clickthrus from each site using parameter tracking in GA. I want to know if I put in a url with ?utm_source=xxx, will this have any impact upon my linkbuilding efforts? i.e. will search engines attribute the keyword to the long url with tracking or the url without tracking. I understand that everything from the ? mark is ignored. However, i just want to double check before I publish release. Thanks for your help. Mik
Reporting & Analytics | | increation0