Bloking pages in roborts.txt that are under a redirected subdomain
-
Hi Everyone,
I have a lot of Marketo landing pages that I don't want to show in SERP. Adding the noindex meta tag for each page will be too much, I have thousands of pages.
Blocking it in roborts.txt could have been an option, BUT, the subdomain homepage is redirected to my main domain (with a 302) so I may confuse search engines ( should they follow the redirect or should they block)
marketo.mydomain.com is redirected to www.mydomain.com
disallow: / (I think this will be confusing with the redirect)
I don't have folders, all pages are under the subdomain, so I can't block folders in Robots.txt also
Would anyone had this scenario or any suggestions?
I appreciate your thoughts here.
Thank you
Rachel
-
Thank you so much for you answer!
the home page in the subdomain is redirected but none of the actual pages in the subdomain are, and because there are so many of them, it would be easier to block them in robots.txt, even if there is small change that Google will still index them. But because the home page is redirected, I don't want to confuse Google with a Disallow: /
Could I do Disallow: / and then Allow: /homepage.html
-
Under usual circumstances, Google won't index redirecting addresses which it considers to be a shallow form of 'doorway' page (Google doesn't like to rank those). If I am reading your post right, no pages on the redirected sub-domain can be visited as they all now redirect. Google should start dropping those old URLs from its index automatically. It's important to note that you should be using 301s to eventually de-index the old URLs. If you're using 302s then you're telling Google that the old pages are only being redirected temporarily and they will return (which could mess with indexation)
If I am reading your post wrong and some pages are still live on the old subdomain and are not redirecting, and you want Google to redirect most addresses but some are staying (for whatever reason) and you still want to de-index those specific ones, Meta no-index (sorry) really is your best bet! Robots.txt tells Google not to crawl a page but it doesn't tell Google not to index a URL if external metrics (inbound links) are strong enough. Both are 'directives' and Google is forced to obey neither
I think from what you are saying, I'd just leave the redirects and let Google do its work. Make sure they're 301s, though. You're right that Robots.txt might end up confusing things. Robots.txt can also sometimes 'kill' the SEO authority of a page. If you did that for all the redirecting pages, no equity would flow through your 301s (hazardous)
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
[Organization schema] Which Facebook page should be put in "sameAs" if our organization has separate Facebook pages for different countries?
We operate in several countries and have this kind of domain structure:
Technical SEO | | Telsenome
example.com/us
example.com/gb
example.com/au For our schemas we've planned to add an Organization schema on our top domain, and let all pages point to it. This introduces a problem and that is that we have a separate Facebook page for every country. Should we put one Facebook page in the "sameAs" array? Or all of our Facebook pages? Or should we skip it altogether? Only one Facebook page:
{
"@type": "Organization",
"@id": "https://example.com/org/#organization",
"name": "Org name",
"url": "https://example.com/org/",
"sameAs": [
"https://www.linkedin.com/company/xxx",
"https://www.facebook.com/xxx_us"
], All Facebook pages:
{
"@type": "Organization",
"@id": "https://example.com/org/#organization",
"name": "Org name",
"url": "https://example.com/org/",
"sameAs": [
"https://www.linkedin.com/company/xxx",
"https://www.facebook.com/xxx_us"
"https://www.facebook.com/xxx_gb"
"https://www.facebook.com/xxx_au"
], Bonus question: This reasoning springs from the thought that we only should have one Organization schema? Or can we have a multiple sub organizations?0 -
Subdomain Question
Having a difficult time on our site and looking for some advice. Our site pages are indexed perfectly, however, we have a subdomain where we have all of our images and PDF's. We only have the main domain set-up in Search Console with our sitemap. We can't seem to get any of our images indexed by Google that are in the subdomain however all the PDF's are indexed. My thought is to add the subdomain to SC and create a new sitemap that is just for the subdomain. Assuming we are not blocking any folders or files with our robots.txt can anyone think of any other reasons why the images wouldn't get indexed.
Technical SEO | | cbathd
Thanks0 -
Duplicate page titles for blog snippets pages
I can't figure the answer to this issue, on my blog I have a number of pages which each show snippets and an image for each blog entry, these are called /recent-weddings/page/1 /2 /3 and so on. I'm getting duplicate page titles for these but can't find anywhere on Wordpress to set a unique title for them. So http://www.weddingphotojournalist.co.uk/recent-weddings/…/2/ has the same title as http://www.weddingphotojournalist.co.uk/recent-weddings/…/3/
Technical SEO | | simonatkinsphoto0 -
Switchboard Tags - Multiple desktop pages pointing to one mobile page
I have recently started to implement switchboard tags to connect our mobile and desktop pages, and to ensure that our mobile pages show up in rankings for mobile users. Because our desktop site is much deeper in content than our mobile site, there are a number of desktop pages we would like to have point to one mobile page. However, with the switchboard tags, this poses a problem because it requires multiple rel=canonical tags to be placed on the one mobile page. I'm assuming this will either confuse the search engines, or they will choose to ignore the rel=canonical tag altogether. Any ideas on how to approach this situation other than creating an equivalent mobile version of every desktop page or implementing a user agent detection redirect?
Technical SEO | | JBlank0 -
Rel canonical for partner sites - product pages only or also homepage and other key pages?
Hello there Our main site is www.arenaflowers.com. We also run a number of partner sites (eg: http://flowershop.cancerresearchuk.org/). We've relcanonical'd the products on the partner site back to the main (arenaflowers.com) site. eg: http://flowershop.cancerresearchuk.org/flowers/tutti_frutti_es_2013 rel canonicals back to: http://www.arenaflowers.com/flowers/tutti_frutti_es_2013). My question: Should we also relcanonical the homepage and other key pages on partner sites back to the main arenaflowers website too? The content is similar but not identical. We don't want our partner sites to be outranking the original (as is the case on kw flower delivery for example). (NB this situation may be complicated by the fact we appear to have an unnatural link penalty on af.com (and when we did an upgrade a while back, the af.com site fell out of the index altogether due to some issues with our move to AWS.) We're getting professional SEO advice on this but wondered what the Moz community's thoughts were.. Cheers, Will
Technical SEO | | ArenaFlowers.com0 -
301 redirects reverting to 302 redirects
We recently built a new website with a new site structure. To prevent there being a load of 404's I redirected the old pages to the new relevant pages with 301 redirects. A few days later the SEOmoz crawl report alerted me to a load of 302 redirects. When I looked into this for some reason all of the 301 redirects I set up are reverting to 302 redirects. I did a test by 301 redirecting a made up URL to an existing page and the same thing happens - it 302 redirects. I can't find any settings in WordPress to possibly explain why this is happening. Has anyone got any ideas why this could be?
Technical SEO | | Tone_Agency0 -
Can you 301 redirect a page to an already existing/old page ?
If you delete a page (say a sub department/category page on an ecommerce store) should you 301 redirect its url to the nearest equivalent page still on the site or just delete and forget about it ? Generally should you try and 301 redirect any old pages your deleting if you can find suitable page with similar content to redirect to. Wont G consider it weird if you say a page has moved permenantly to such and such an address if that page/address existed before ? I presume its fine since say in the scenario of consolidating departments on your store you want to redirect the department page your going to delete to the existing pages/department you are consolidating old departments products into ?
Technical SEO | | Dan-Lawrence0 -
Redirecting internal pages to another domain
Hi, I recently redirected the domain www.organic7thheaven.com to http://www.7thheavennaturals.com/ However, the internal pages of www.organic7thheaven.com such as http://www.7thheavennaturals.com/products/sensitivesouls/ are now displaying a 404 Not Found Message How can I have it so that pages such as the above automatically also redirect to http://www.7thheavennaturals.com/ Thanks!
Technical SEO | | MJMarketing0