Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
-
Hi all,
The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong.
Thanks
-
I agree with Gaston's approach right up to step 4. If you add the no-indexed pages back into a block in the robots.txt file, you'll end up back where you started from. Because Google will still discover the no-indexed URLs elsewhere and the robots,txt block will stop them from discovering the no-index, and the URLs will likely start to get added to the index again.
No-indexed URLs must not be blocked in robots.txt. Those two processes are mutually exclusive.
-
Hi there,
TLDR; The solution to deindexing and never index again:
- Allow (with robots.txt) the web to be crawable
- Aplly meta robots tag: noindex,follow
- Wait somte weeks to be completely deindexed
- block the entire site/section with robots.txt
Robots.txt and the robots meta tag can make the same effect, but to understand them must be analyzed separatedly.
-
Robots.txt, here you just tell bots where they can go BEFORE they crawl any of the website. This is just a signal, not a directive... Because robots can choose to ignore the what's in the file. Here you can block from the entire web, to an entire section or just specific pages. More info: Robots.txt official page and a really cool and complete guide to robots.txt
-
Robots meta tag, with it you have more signals to tell, the most used are: noindex, nofollow and follow, due to the usual issues about indexing. More info: Robots.txt offical page, Google developers, Meta Robots directive - Moz and a complete guide to meta robots tag - YOAST.
Hope this is what you wanted.
Best luck
GR.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta Keywords is relevant in SEO yet ?
Hello, meta keywords is relevant to use in SEO yet? If yes, how many words is recommended to use? Regards, Leandro
Algorithm Updates | | lmoraes0 -
Content strategy for landing pages: Topics vs Features
Hi all, We are going to create new landing pages and optimise existing pages. We have a confusion on how to employ content on these pages....whether these will be filled with content to rank for "topics" and "keywords" or direclty jump into the features are are providing. If we go with first, users may feel boring about teaching them about that topic, if we go with latter...it's hard to rank being no related content to rank for that topic. I have seen some of the websites are employing multiple landing pages where they fill with topic related content and then link to features pages. I need suggestions here. Thank you
Algorithm Updates | | vtmoz1 -
URL in SERP: Google's stand
Months back, we can notice "keyword" will be bold and highlighted if its in the SERP URL. Now Google no more highlights any URLs even with exact match of keyword we search. Beside UI, Does this mean Google might devalued or reduced the importance of URL as ranking factor? We can see many search results match partially or completely in URL with search keywords.
Algorithm Updates | | vtmoz0 -
Product Listing Pages
Hi I had a question regarding product pages and the best way to display the page for SEO. For example, is it best to have a page for - Blue Euro Containers including a table of the capacity options you can buy.. Or, have each product split out so it has it's own product page - 60L Blue Euro Container, etc etc I know a lot of the information will be fairly similar, with the capacity being the one major difference - is this a bad thing? Some of our product tables are too big and the idea was to split them out. Thanks!
Algorithm Updates | | BeckyKey0 -
Should we use brand name of product in URL
Hi all, What is best for SEO. We sell products online. Is it good to mention the brand in the product detail page URL key if (part of) the brand is also in the home url? So our URL is: www.brandXstore.com Is it best to do: www.brandXstore.com/brandX-productA.html of just do: www.brandXstore.com/ProductA.html Thanks for quick answering 😉
Algorithm Updates | | RetailClicks1 -
Does Google or Bing use words in the page title beyond the displayed limit for ranking purposes?
Standard good practice for on-page SEO includes keeping page title length below the maximum that Google displays in the SERPs. But words in the title beyond that maximum can be indexed, even if they don't show in the SERPs for end users. For ranking purposes, is there any value in words beyond the character limit in page titles that are truncated in the SERPs?
Algorithm Updates | | KyleJB0 -
Did we get hit by Panda? What do we do?
Hello, here's our site: nlpca(dot)com We had a big drop in rankings, going from about 19th to about 43rd for our main keyword and having significant drops in other keywords. This happened roughly 6 weeks ago We thought it was being caused by either: Placing keywords in titles before we had them in the content. or Trying to rank for Utah keywords - we're the NLP Institute of California and we are in both places now, but the site talks about mainly California. We changed both these things, and we're still at the low rankings. Will we move back up? What do we do? Will a backlink campaign be effective at this point?
Algorithm Updates | | BobGW0