Bloking pages in roborts.txt that are under a redirected subdomain
-
Hi Everyone,
I have a lot of Marketo landing pages that I don't want to show in SERP. Adding the noindex meta tag for each page will be too much, I have thousands of pages.
Blocking it in roborts.txt could have been an option, BUT, the subdomain homepage is redirected to my main domain (with a 302) so I may confuse search engines ( should they follow the redirect or should they block)
marketo.mydomain.com is redirected to www.mydomain.com
disallow: / (I think this will be confusing with the redirect)
I don't have folders, all pages are under the subdomain, so I can't block folders in Robots.txt also
Would anyone had this scenario or any suggestions?
I appreciate your thoughts here.
Thank you
Rachel
-
Thank you so much for you answer!
the home page in the subdomain is redirected but none of the actual pages in the subdomain are, and because there are so many of them, it would be easier to block them in robots.txt, even if there is small change that Google will still index them. But because the home page is redirected, I don't want to confuse Google with a Disallow: /
Could I do Disallow: / and then Allow: /homepage.html
-
Under usual circumstances, Google won't index redirecting addresses which it considers to be a shallow form of 'doorway' page (Google doesn't like to rank those). If I am reading your post right, no pages on the redirected sub-domain can be visited as they all now redirect. Google should start dropping those old URLs from its index automatically. It's important to note that you should be using 301s to eventually de-index the old URLs. If you're using 302s then you're telling Google that the old pages are only being redirected temporarily and they will return (which could mess with indexation)
If I am reading your post wrong and some pages are still live on the old subdomain and are not redirecting, and you want Google to redirect most addresses but some are staying (for whatever reason) and you still want to de-index those specific ones, Meta no-index (sorry) really is your best bet! Robots.txt tells Google not to crawl a page but it doesn't tell Google not to index a URL if external metrics (inbound links) are strong enough. Both are 'directives' and Google is forced to obey neither
I think from what you are saying, I'd just leave the redirects and let Google do its work. Make sure they're 301s, though. You're right that Robots.txt might end up confusing things. Robots.txt can also sometimes 'kill' the SEO authority of a page. If you did that for all the redirecting pages, no equity would flow through your 301s (hazardous)
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to Choose destination page for a 301 redirect?
I am doing some SEO for a wedding chapel in Vegas. There are some old packages that no longer exist and the bounce rate for the page is high so I am planning to 301 the page. How to best determine the best 301 destination? I have a few options. As an example the page was optimized for garden weddings. The page itself does not place well in the SERPS for garden weddings in Las Vegas, but our outdoor wedding packages in Las Vegas page places in the top 10. So that page is in an option. However, there is a different location that has a garden setting. Is that a better choice? Some content might match better than others, but any page I choose would be relevant content. Thank you so much 🙂
Technical SEO | | leslieevarts0 -
Moving Some Content From Page A to Page B
Page A has written content, pictures, videos. The written content from Page A is being moved to Page B. When Google crawls the pages next time around will Page B receive the content credit? Will there not be any issues that this content originally belonged to Page A? Page A is not a page I want to rank for (just have great pictures and videos for users). Can I 301 redirect from Page A to B since the written content from A has been deleted or no need? Again, I intent to keep Page A live because good value for users to see pictures and videos.
Technical SEO | | khi50 -
Robots.txt
Hello, My client has a robots.txt file which says this: User-agent: * Crawl-delay: 2 I put it through a robots checker which said that it must have a **disallow command**. So should it say this: User-agent: * Disallow: crawl-delay: 2 What effect (if any) would not having a disallow command make? Thanks
Technical SEO | | AL123al0 -
Canonical redirects
Hello, I have a quick question: I use wordpress for my website. I have a plugin for translating the website in other languages. Thus, I have 2 versions of urls, one with /en, one without (original languale). This has been seen as duplicate content. I have been advised that the best to do is to use canonical redirect. Should I use it on the general header.php (the only header I can find in the CMS), or should I redirect each page singularly? I believe the second is the best way, but I can't find headers and txt documents for each page in my FTP. As well I have seen this post, in which is explained that canonical redirects can be done directly in the general header.php http://www.bin-co.com/blog/2009/02/avoid-duplicate-content-use-canonical-url-in-wordpress-fix-plugin/ Is it true? You have any suggestion?
Technical SEO | | socialengaged
Thanks! 🙂 Eugenio0 -
301 Redirect From Dynamic Page To Static
I want to 301 redirect all "id" and "type" numbers from my page dynamic.php page (I have thousands of IDs and thousands of Types) all to a single URL. So for example the following.... www.mysite.com/dynamic.php?id=1&type=5 www.mysite.com/dynamic.php?id=2&type=5 www.mysite.com/dynamic.php?id=3&type=5 www.mysite.com/dynamic.php?id=1&type=6 www.mysite.com/dynamic.php?id=2&type=6 www.mysite.com/dynamic.php?id=3&type=6 ...would all be sent to: www.mysite.com/page.html How can this be done without doing a redirect for each ID/Type?
Technical SEO | | TheDude1 -
How do I redirect non www pages to www on a windows server?
As the .htaccess file cannot be worked on, I added this php code 301 redirect if the URL does not contain a www on all the pages (small website - 10 pages) : header( "HTTP/1.1 301 Moved Permanently" ); header( "Location: $location" ); I want to know if this is ok for SEO? Has anyone done this on a windows server? Or if you have any better methods, it would be great if you can share. Please help. Thanks.
Technical SEO | | ArjunRajkumar0 -
Discrepency between # of pages and # of pages indexed
Here is some background: The site in question has approximately 10,000 pages and Google Webmaster shows that 10,000 urls(pages were submitted) 2) Only 5,500 pages appear in the Google index 3) Webmaster shows that approximately 200 pages could not be crawled for various reasons 4) SEOMOZ shows about 1,000 pages that have long URL's or Page Titles (which we are correcting) 5) No other errors are being reported in either Webmaster or SEO MOZ 6) This is a new site launched six weeks ago. Within two weeks of launching, Google had indexed all 10,000 pages and showed 9,800 in the index but over the last few weeks, the number of pages in the index kept dropping until it reached 5,500 where it has been stable for two weeks. Any ideas of what the issue might be? Also, is there a way to download all of the pages that are being included in that index as this might help troubleshoot?
Technical SEO | | Mont0 -
Too many on page links for WP blog page
Hello, I have set my WP blog to a page so new posts go to that page making it the blog. On a SEOmoz campaign crawl, it says there are too many links on one page, so does this mean that as I am posting my blog posts to this page, the search engines are seeing the page as one page with links instead of the blog posts? I worry that if I continue to add more posts (which obviously I want to) the links will increase more and more, meaning that they will be discounted due to too many links. What can I do to rectify this? Many thanks in advance
Technical SEO | | mozUser14692366292850