Get a list of robots.txt blocked URL and tell Google to crawl and index it.
-
Some of my key pages got blocked by robots.txt file and I have made required changes in robots.txt file but how can I get the blocked URL's list.
My webmaster page Health>blocked URL's shows only number not the blocked URL's.My first question is from where can I fetch these blocked URL's and how can I get them back in searches,
One other interesting point I see is that blocked pages are still showing up in searches.Title is appearing fine but Description shows blocked by robots.txt file.
I need urgent recommendation as I do not want to see drop in my traffic any more.
-
"changing the lastmod of those pages to today".
How can I make these changes?
Right now the news is that Resubmitted the Sitemap and no warnings this time.
-
I imagine that since you've got a robots txt error you'll probably ended closing a whole directory to bots which you wanted to be indexed. You can easily spot the directory and resubmit a sitemap to google changing the lastmod of those pages to today and the priority to 1 but only of those pages.
If you still receive warnings it may be due to errors in your sitemap. You're probably including some directory you don't want. You can try it in GWT putting in the box at the bottom the url you want to maintain in the index and then trying to see if some urls are being blocked by your robots.
If you want you can post here your robots and the URIs you want to be indexed without knowing the domain so that won't be public. Hope this may help you
-
Ok Resubmitted it.but even with updated file it gives a lot of errors.I think it takes some time.20,016 warnings
I have not added no index attribute in my header region.It was all messy stuff with robots.txt file.It means that with site showing up in SERP the rank will probably be the same or it was deranked?
-
Go into GWMT and resubmit sitemap.xml files (with the URLs you want indexed) for recrawling and Google will digest the sitemaps again, instead of waiting for Googlebot to come around on their own, you are requesting it to come around, also include those new sitemap files in your robots.txt file.
-
In Google Webmaster Tools, go to Health -> Fetch As Google. Then add the previously blocked URL and click Fetch. Once you've done that, refresh the page and click "Submit to index". That should get Google indexing those pages again.
Getting external links to your pages also helps get pages crawled & indexed, so it may be worth submitting your pages to social bookmarking sites, or get other types of backlinks to your previously blocked pages if possible.
-
Since you fixed your robots.txt file you should be good to. It will probably take a few days for Google to recrawl your site and update the index with the URLs they are now allow to crawl.
Blocked URLs can still show up in SERPs if you haven't defined the no-index attribute in your section.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Console not loading some resources
When I check an URL with Search Console it cannot load some page resources, even from other domains (like: ssl.google-analytics.com, www.facebook.com and www.google-analytics.com).
Reporting & Analytics | | TottiataHUN
Have any of you experienced this issue?
Steps to reproduce: open Google Search Console check an URL click "View crawled page" link select "More info" tab click "Page resources" {?}/{?} couldn't be loaded When I check the listed resources, all of them can be loaded from a web browser.
So I do not understand why Google cannot load them.
And there is no additional info why the resources could not be loaded.
Any ideas? google-search-console-other-error-1.png google-search-console-other-error-2.png0 -
"index.htm" for all url's in google analytics
I don't have this issue with other wordpress websites, only this one website, and I don't know what's causing the issue: Google Analytics is adding an "index.htm" to every single page on the website. So it is tracking the pages, I see no errors - is it tracking the right page? When I click on the page link in a report, I naturally go to a "404 page not found" since the website address isn't "www.example.com/rewards/index.htm" - but instead the actual address would be:
Reporting & Analytics | | cceebar
"www.example.com/rewards/". I have navigated to View Settings in GA to insure "default page" is empty. Although adding anything else to this field does not effect the page url in analytics reports either. Could it be htaccess file - or a plugin effecting the htaccess file?_Cindy0 -
Large event site - how should I structure my URLs?
Hi guys, I'm working on a new website which is consolidating a number of existing event sites into one. The existing sites use a variety of URL structures: www.eventsite1.com/events/event-name www.eventsite2.com/festival-program/event-name www.eventsite3.com/event-name This inconsistency has led to issues with tracking category usage properly in analytics - for instance, with eventsite3.com, events fall within categories (www.eventsite3.com/category-name) but as soon as you drill into an event detail page (www.eventsite3.com/event-name) from the category page, the category is lost to analytics. This is compounded when one event lives within multiple categories, as I can't figure out which category is the most effective for a particular event. I've seen other event sites establish a canonical URL for a primary category, display it in the URL (i.e. www.eventsite4.com/primary-category/event-name) yet still let that event get hit via the secondary categories (www.eventsite4.com/secondary-category/event-name). This way, the categories get passed to analytics without any duplicate content issues (i.e. via the setting of canonicals) Basically, I want to make sure that whatever instruction I give to the devs for the new site re: URL structure is correct from an SEO perspective and analytics perspective. Do I even need to worry about having the category in the URL? Can someone please help me with this? Hope this makes sense Cheers
Reporting & Analytics | | cos20300 -
Double click by Google - whats that all about?
Buongiorno from Latitude 53.92705600 Longitude -1.38481600... Ok ive watched the video, ive read the blog posts but what is Double click by Google? how do i get to use it?
Reporting & Analytics | | Nightwing
http://doubleclickadvertisers.blogspot.co.uk/2012/06/digital-marketing-platform-to-win.html Is it a seperate service to Adwords? Questions questions... Any insights welcome 🙂0 -
Is Google analytics not working properly anymore?
Hi mozzers, We launched a website 2 months ago and when monitoring Organic data on GA we had an average of 23% bounce rate like a month ago but the last 2 weeks I noticed that bounce rate=0%. This is very weird especially that we haven't modified the existing pages and that multiple keywords show an average site duration of 0 seconds. Any thoughts about this discrepancy? Do you think that there is an issue with the GATC implemented? Thanks
Reporting & Analytics | | Ideas-Money-Art0 -
Google Maps not passing referral data
Google Maps is not passing referral data (URLs, not KWs). Google+ Local is referring, but nothing from maps. Maps referrals appear to be coming across as direct. Any ideas? We haven't found anything online, one of the guys at the office documented what we did find, using Chrome's debugger - http://manofactionmetrics.com/2012/11/02/google-maps-not-passing-any-referral-data/
Reporting & Analytics | | Danieljacobree0 -
Where are google analytics stats for iphone4
hi We were looking at the Google Analytics for one of our sites and noticed that there were NO pageviews from device=iphone and resolution=640x960 in the report. Given that iphone4 is supposed to be 640x960, and would be the most popular device (at least in our offices and everyone I know), it seems wierd. I sorted the Mobile Devices report by device and resolution to see what was available. The first 160 results were all device=not set. Finally got to device=iPhone and there were three entries: resolution 0x0 had 11 views resolution 320x396 had 45 views resolution 320x480 had 3,944 views. Hopefully all iphone4 users havent been classified as not set. Or is it possible that iphone4s claim to be 320x480 in browsers, as per http://www.alistapart.com/articles/a-pixel-identity-crisis/ Even worse, if I look at the Samsung Galaxy S II (myown phone), there are over 30 screen resolution combinations. Does anyone have anything to shed on this? I asked about it on the google analytics twitter account last week but havent had a response. Are there other analytics solutions that would distinguish between the iphones? Warning - this is a link to a large image, with the not set stats at the top. 6Sjji
Reporting & Analytics | | ozgeekmum0 -
Google and bing search filed commands
Dose someone have / know a full list / resource with commands for google and bing ? Including filters for those commands ? (site:domain.com -filter etc) (like: site:domain.com, link:domain.com etc) I use the basic ones b ut I know there are much more and that there are several filters that can be used with success to filter down results. Thanks.
Reporting & Analytics | | eyepaq1