How do I disallow crawl on a directory when it's a prefix to my site's URL?
-
I am trying to disallow our media repository (hosted elsewhere, but appears as a directory on our site) from being crawled by robots but it is not a subdirectory of the site, it's a prefix.
So I need to disallow: mediabank.mywebsite.org
Not: mysite.org/mediabank
What would I need to put in my robots.txt and/or the other host's robots.txt to make this happen?
Thanks!
-
Hey there! Tawny from Moz's Help Team here.
You'll want to add a robots.txt file for that subdomain, and then add a Disallow command to that robots.txt file. So, using your example, you'd want a file like mediabank.mywebsite.org/robots.txt that had a Disallow command for any robots you don't want crawling that subdomain.
For all user-agents, that would look something like this:
User-agent: *
Disallow: /That would stop any user-agents from crawling any pages on that subdomain.
I hope this helps! If you've still got questions, feel free to send us a note at [email protected] and we'll do our best to sort things out for you.
-
Hi,
Please check this old thread on the same topic @ https://mza.seotoolninja.com/community/q/block-an-entire-subdomain-with-robots-txt
Thanks
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl Issue Question
Hey guys, I have run the crawl on my WordPress site and Moz finds a "Critical crawl issue" for my site on a broken link (404 error): mydomain.com/**%25s **, I can't seem to be able to find such a link anyway and I have run the website through several other tools that scan for broken links and such and there is no such result.
Moz Bar | | K.Net
This link doesn't exist on my site at all and I don't know where Moz got it from, I have made changes to my site and recrawled several times and the specific error persists. Does anyone have any ideas?0 -
Spam site
my website is activate in egg incubation industry https://www.taksafir.com . Level of this site spam score is 11% . now how can i reduce that ?
Moz Bar | | HeidiMaryAyuningtyas0 -
Only One Canonical URL Tag
HI, I'm an SEO novice - company owner with no money so doing it all myself with help from my web designer using wordpress. Ive just completed some seo and done the moz page scoring analysis for optimisation and gained 92% - however - there is one outstanding issue on canonical url tags - i.e. recommened fix = The canonical URL tag is intended to refer duplicate pages to a single canonical URL. To ensure the search engines properly parse the canonical source, your page should use only one version of this tag in the header. See Canonical URL Tag - the Most Important Advancement in SEO Practices Since Sitemaps Ive gone through the page code and can see I have 2 rel=canonical references - am I able to simply delete one - how do I do this if its been created by the yoast/wordpress plug-in? Many thanks in advance for any help!
Moz Bar | | M-J-Smith0 -
Error in Duplicate Content Being Reported - Pages Aren't Actually Duplicates
The recent crawl of one of our sites revealed a high number of duplicate content issues. However, when I viewed the report for pages with duplicate content I noticed almost all of them are not duplicates. For example, these two pages are marked as dupes:
Moz Bar | | M_D_Golden_Peak
https://www.writersstore.com/publishers/hollywood-creative-directory
https://www.writersstore.com/authors/g-miki-hayden These are thin as far as content goes but definitely not duplicates. Any recommendations or ways to adjust the settings so that these false positives aren't clogging up our site crawl report?0 -
How much time should I wait between Crawl Tests?
Hello! I ask because it has happened before (and again this morning) that after doing a crawl test and repairing my site per the errors found in Moz's crawl test it still finds the same error. Even though I fixed them. Typically I do a re-crawl 6 hours after or the next day and I find the same errors. I know they are fixed because a couple of days go by and finally Moz gets it right. I had understood that the crawl test was an "on-demand" crawl of sorts, granted with limit of 2 a day. But it seems that if you re-crawl your site within a day the same results yield? It's frustrating. Is this correct? Thank you!
Moz Bar | | md30 -
I'm checking keyword difficulty for two different sites. Would love to view the results by site instead of just one large list. Is that possible? Or would it just be easier to keep the lists separate in Excel and just import when I want an updated report?
I have keyword lists for two sites. Is there a way to label them in the keyword difficulty tool (List A, List B) so I can just view results for a particular site? Or do I need to run the report with List A, export results, delete those keywords, then run the report for List B?
Moz Bar | | JohnNovakLV0 -
Rank Checker Won't Accept New gTLDs
Hi everyone, I've got some domains with extension** .solutions** however, these extensions are not yet accepted by some of the very useful, and now dearly missed tools on this site. One of those tools is the Rank Checker: error message TDanTx1.png
Moz Bar | | SSsseeeooOO0 -
Site Crawler Tool by the Company Formerly Known As SEOMoz
Moz had a tool I used that would crawl my site and send me a report of all pages, all errors, 301s 404s 505s, and a whole plethora of stuff. I used it to fix pesky errors quite a bit. Does this still exist? Was it replaced or am I just not finding it in the new design?
Moz Bar | | KJ-Rodgers0