Get a list of robots.txt blocked URL and tell Google to crawl and index it.
-
Some of my key pages got blocked by robots.txt file and I have made required changes in robots.txt file but how can I get the blocked URL's list.
My webmaster page Health>blocked URL's shows only number not the blocked URL's.My first question is from where can I fetch these blocked URL's and how can I get them back in searches,
One other interesting point I see is that blocked pages are still showing up in searches.Title is appearing fine but Description shows blocked by robots.txt file.
I need urgent recommendation as I do not want to see drop in my traffic any more.
-
"changing the lastmod of those pages to today".
How can I make these changes?
Right now the news is that Resubmitted the Sitemap and no warnings this time.
-
I imagine that since you've got a robots txt error you'll probably ended closing a whole directory to bots which you wanted to be indexed. You can easily spot the directory and resubmit a sitemap to google changing the lastmod of those pages to today and the priority to 1 but only of those pages.
If you still receive warnings it may be due to errors in your sitemap. You're probably including some directory you don't want. You can try it in GWT putting in the box at the bottom the url you want to maintain in the index and then trying to see if some urls are being blocked by your robots.
If you want you can post here your robots and the URIs you want to be indexed without knowing the domain so that won't be public. Hope this may help you
-
Ok Resubmitted it.but even with updated file it gives a lot of errors.I think it takes some time.20,016 warnings
I have not added no index attribute in my header region.It was all messy stuff with robots.txt file.It means that with site showing up in SERP the rank will probably be the same or it was deranked?
-
Go into GWMT and resubmit sitemap.xml files (with the URLs you want indexed) for recrawling and Google will digest the sitemaps again, instead of waiting for Googlebot to come around on their own, you are requesting it to come around, also include those new sitemap files in your robots.txt file.
-
In Google Webmaster Tools, go to Health -> Fetch As Google. Then add the previously blocked URL and click Fetch. Once you've done that, refresh the page and click "Submit to index". That should get Google indexing those pages again.
Getting external links to your pages also helps get pages crawled & indexed, so it may be worth submitting your pages to social bookmarking sites, or get other types of backlinks to your previously blocked pages if possible.
-
Since you fixed your robots.txt file you should be good to. It will probably take a few days for Google to recrawl your site and update the index with the URLs they are now allow to crawl.
Blocked URLs can still show up in SERPs if you haven't defined the no-index attribute in your section.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Google Analytics (GA4) recommendations for SEO analysis?
Guides on Moz and elsewhere mostly refer to Google Analytics' Universal Analytics (UA). However, UA is being replaced with GA4, and the interface, options, and reporting are very different. Can you recommend a clear, thorough, and effective walkthrough of how to set up useful SEO reports in GA4? Is there a simple tool you recommend that will help connect historical data from UA to GA4 when GA4 is the only option available? If there's no simple tool, what values do you recommend retaining from UA for effective historical reporting? How would you use them? At minimum for reporting, I'd want to show month-to-month changes and year-to-year changes (in percentages and in real numbers) for the following: all site visits all organic visits organic visits as a percentage of all site visits organic visits that led to a specific goal completion organic visits that led to any goal completion Thanks in advance for your help!
Reporting & Analytics | | Kevin_P1 -
Google Analytics & Google Sheets
Hi guys, I'm looking to use google analytics google sheets extension to pull data into a google sheet with metrics from other tools e.g. semrush, moz, etc. From my understanding with google analytics google sheet extension its designed to run reports and doesn't really allow you to add other metrics. Basically what i'm trying to do is this: https://www.useloom.com/share/c1e42bfa60bd46fca2b1120018969ce8 Any suggestions/advice on how to do this would be great. Cheers.
Reporting & Analytics | | jayoliverwright0 -
Dynamic URL parameter generator
Hi Is any one privy to an online tool that will let me create a dynamic URL parameter string but will allow me to generate multiples of the URL and add a distinct key for each one. E.g Campaign Source, Medium, Name, a Keywords etc. are all the same in the string but then I want to generate a unique id code at the end. Then export them as a csv and integrate into my database lists. Looking to run this into a few thousand as well. I was going to just do this in Excel and combine two columns withe the string and the number count in the other column but if there is a tool that does it all that would be interesting to know.
Reporting & Analytics | | David-E-Carey0 -
Google Analytics - how do you find out Unique Visitors ?
Hi Im trying to find out unique visitors per annum in GA, is this possible, is it simply users ? i know they changed definitions recently cheers dan
Reporting & Analytics | | Dan-Lawrence0 -
Google Analytics
How can you see the conversion rates for the referral traffic? For example I want to see if any of the facebook visits to my site resulted in a purchase.
Reporting & Analytics | | EcomLkwd0 -
Setting up goals - google analytics
Hi Im new user google analytics I would like to set up goal for website. I asked to basic questions: What are my bussiness objectives? Answer:make the visitor to click on a advertisment (affiliate links and affiliate banners) to be redirected to company website to finish purchase. How to set up goals? How can I track ads (affiliate banners and affiliate links) on my website?
Reporting & Analytics | | info_tipovanie-stavkovanie.com0 -
Links and index missing from Google Webmaster Tools!
I've been using a wordpress blog and using google analytics and webmaster tools. I'd never done very much optimizing of my site, my strategy had always been to just keep writing relevant posts and let google do the rest. At some point I installed a SEO plugin to wordpress and did a tiny amount of tweaking, but I noticed that in my webmaster tools page, all my "links to this site" links are gone! My site map says 42 pages submitted, 42 pages indexed, but all the links that I've built over the past few years are missing, and I also notice that I can't find my main landing page in the index anymore. I know there are indexed sites out there that link to me, but they aren't being reflected anymore. Did I anger google by using SEO plugins within wordpress? I'm fairly certain I wasn't doing anything sketchy. Http://brodiegraphics.com RTOH6.png
Reporting & Analytics | | HELPeR0 -
Google and bing search filed commands
Dose someone have / know a full list / resource with commands for google and bing ? Including filters for those commands ? (site:domain.com -filter etc) (like: site:domain.com, link:domain.com etc) I use the basic ones b ut I know there are much more and that there are several filters that can be used with success to filter down results. Thanks.
Reporting & Analytics | | eyepaq1