Get a list of robots.txt blocked URL and tell Google to crawl and index it.
-
Some of my key pages got blocked by robots.txt file and I have made required changes in robots.txt file but how can I get the blocked URL's list.
My webmaster page Health>blocked URL's shows only number not the blocked URL's.My first question is from where can I fetch these blocked URL's and how can I get them back in searches,
One other interesting point I see is that blocked pages are still showing up in searches.Title is appearing fine but Description shows blocked by robots.txt file.
I need urgent recommendation as I do not want to see drop in my traffic any more.
-
"changing the lastmod of those pages to today".
How can I make these changes?
Right now the news is that Resubmitted the Sitemap and no warnings this time.
-
I imagine that since you've got a robots txt error you'll probably ended closing a whole directory to bots which you wanted to be indexed. You can easily spot the directory and resubmit a sitemap to google changing the lastmod of those pages to today and the priority to 1 but only of those pages.
If you still receive warnings it may be due to errors in your sitemap. You're probably including some directory you don't want. You can try it in GWT putting in the box at the bottom the url you want to maintain in the index and then trying to see if some urls are being blocked by your robots.
If you want you can post here your robots and the URIs you want to be indexed without knowing the domain so that won't be public. Hope this may help you
-
Ok Resubmitted it.but even with updated file it gives a lot of errors.I think it takes some time.20,016 warnings
I have not added no index attribute in my header region.It was all messy stuff with robots.txt file.It means that with site showing up in SERP the rank will probably be the same or it was deranked?
-
Go into GWMT and resubmit sitemap.xml files (with the URLs you want indexed) for recrawling and Google will digest the sitemaps again, instead of waiting for Googlebot to come around on their own, you are requesting it to come around, also include those new sitemap files in your robots.txt file.
-
In Google Webmaster Tools, go to Health -> Fetch As Google. Then add the previously blocked URL and click Fetch. Once you've done that, refresh the page and click "Submit to index". That should get Google indexing those pages again.
Getting external links to your pages also helps get pages crawled & indexed, so it may be worth submitting your pages to social bookmarking sites, or get other types of backlinks to your previously blocked pages if possible.
-
Since you fixed your robots.txt file you should be good to. It will probably take a few days for Google to recrawl your site and update the index with the URLs they are now allow to crawl.
Blocked URLs can still show up in SERPs if you haven't defined the no-index attribute in your section.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Link juice am i getting it or not?
We have a newspaper source that links to a domain that forwards to a page on our website. Do we get link juice from that? Go here: https://dailygazette.com/ click on "Classifieds" then "CapRegion Homes" the button goes to the website www.capregionhomes.com which forwards to our main website, a page we built for them. My question is: Because the link from this site first goes to another site that auto-forwards to us are we actually getting any link juice from the daily gazette on this or is that first site getting all the juice & we technically are getting link juice from capregionhomes.com ?
Reporting & Analytics | | Cfarcher0 -
Dynamic URL parameter generator
Hi Is any one privy to an online tool that will let me create a dynamic URL parameter string but will allow me to generate multiples of the URL and add a distinct key for each one. E.g Campaign Source, Medium, Name, a Keywords etc. are all the same in the string but then I want to generate a unique id code at the end. Then export them as a csv and integrate into my database lists. Looking to run this into a few thousand as well. I was going to just do this in Excel and combine two columns withe the string and the number count in the other column but if there is a tool that does it all that would be interesting to know.
Reporting & Analytics | | David-E-Carey0 -
Tracking Google places (7 pack listing) traffic in google analytics
Is there a way to see Google Places traffic (traffic from users clicking through the 7 pack listings) segmented in Google analytics ? Normally is it just lumped together with the organic traffic ? Can you see the search phrases used to find your site, or do they also show up under 'not provided' when from Google Places. Im aware i can see some limited data in the Google Places analytics, but these seem to be 2 days behind when ever i view them.
Reporting & Analytics | | Sam-P0 -
Getting google impressions for a site not in the index...
Hi all Wondering if i could pick the brains of those wise than myself... my client has an https website with tons of pages indexed and all ranking well, however somehow they managed to also set their server up so that non https versions of the pages were getting indexed and thus we had the same page indexed twice in the engine but on slightly different urls (it uses a cms so all the internal links are relative too). The non https is mainly used as a dev testing environment. Upon seeing this we did a google remove request in WMT, and added noindex in the robots and that saw the index pages drop over night. See image 1. However, the site still appears to getting return for a couple of 100 searches a day! The main site gets about 25,000 impressions so it's way down but i'm puzzled as to how a site which has been blocked can appear for that many searches and if we are still liable for duplicate content issues. Any thoughts are most welcome. Sorry, I am unable to share the site name i'm afraid. Client is very strict on this. Thanks, Carl image1.png
Reporting & Analytics | | carl_daedricdigital0 -
Keeping Google Analytics Data when Moving to Subdomain
Hey All, Against my objections a client has decided to move an existing site into a subdomain while putting up a new site on the main domain. My question revolves around Google Analytics, how do I make sure that I don't lose historical data on the domain before it moves to a subdomain? We're going to be doing a redesign of the old site and I need to keep the historical data so I can prioritize content. What do I need to do? Or will Google analytics recognize the URL's and still attribute the data to those URL's or will I have a separate set of data based on the new URL (with the subdomain). Any insight would be appreciated! Thanks!
Reporting & Analytics | | EvansHunt0 -
Accidental Link not being removed by Google WMT
I operate two sites for a client. One is a local business and one is their national business. I used the same template for both sites (with changes) but accidentally left a link in the footer to the local site. Now the local site is showing 12k backlinks from the national site. I removed the link over 2 weeks ago but it still shows up in Google WMT in the "Links to your Site" section. It goes to a coupon page and not a "targeted" page but 12k links to the local site is 6 TIMES what I had before. My question is: "Is there a way to get Google to remove the link from Google WMT?" More specifically force it. Like I said the link has been removed for over 2 weeks but it still shows up in the Local site's Incoming Links section of WMT. Thanks.
Reporting & Analytics | | DarinPirkey0 -
Google encryption of search results
Hello Is there any definitive information on whether the recent changes to the way Google encrypts search results for people logged into Google affects the traffic present via the Google Adwords Keyword tool? Plus, how does SEOmoz ensure/minimise the affect of personalisation/localisation etc. on the rankings data provided via the pro tool. Thanks in advance for your assistance. Kind regards Neil
Reporting & Analytics | | mccormackmorrison0 -
Phantom urls causing 404
I have a very strange problem. When I run SEOmoz diagnostics on my site, it reveals urls that I never created. It seems to combine two slugs into a new url. For example, I have created the pages http://www.naplesrealestatestars.com/abaco-bay-condos-naples/ and http://www.naplesrealestatestars.com/beachwalk-naples-florida/ and now the url http://www.naplesrealestatestars.com/abaco-bay-condos-naples/beachwalk-naples-florida/ exists in addition to the two I created. There are over 100 of these phantom urls and they all show a 404 error when clicked on or crawled by SEOmoz. Any body know how to correct this?
Reporting & Analytics | | DanBoyle760