Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to [email protected] and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz crawler not working
Hi Moz crawler keep failing on my site with the error showing : Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. I'm not sure what am I missing out.. this is my robots.txt.. i don't think Im missing anything else.. https://www.wearefutureheads.com/robots.txt can the support team help ?
Moz Pro | | teikh0 -
Why is Moz Reporting as Duplicate Page Titles?
Our most recent MOZ crawl campaign is reporting 931 duplicate page title errors, most of which are "Product Review" pages like the following. Although there is only one review on this page, http://www.audiobooksonline.com/Cell_Stephen_King_unabridged_compact_discs.html, MOZ is reporting 15 duplicate page title, four of which I present below. http://www.audiobooksonline.com/reviews/review.php/full/0743554337/0/name/desc
Moz Pro | | lbohen
http://www.audiobooksonline.com/reviews/review.php/full/0743554337/0/rating/asc
http://www.audiobooksonline.com/reviews/review.php/full/0743554337/0/rating/desc
http://www.audiobooksonline.com/reviews/review.php/full/0743554337/0/state/asc Why is MOZ reporting these "pages" as duplicate page title errors? Are these errors hurting our SEO? How to fix?0 -
Is it possible to change the order of my campaigns?
I've just upgraded from a PRO to a PRO PLUS account (as if I need more caffine!) and wondered if it were possible to change the order of the campaigns? They seem to be in alphabetical order - which is no problem, but I'm not actually running all the campaigns - I'm only monitoring a few potential sites, so don't want to see them at the top? it's only a niggle, not a gripe, but wondered it it were possible - or if not, can I show more than 5 campaigns at once rather than scrolling. As always, thanks in advance. G.
Moz Pro | | Gordon_Hall0 -
Can I set Up Multiple Campaigns For Same Website?
I would like to set up one campaign for my main site example.com and another campaign for example.com/blog. Is it possible? I don't want crawling to overlap for these two separate campaigns.
Moz Pro | | jombay0 -
Changing Date Range in SEOMoz Campaign Reports?
Hey, I'm trying to create some old reports for a client and wanted to see how I can adjust the reports to a specific timeframe so I can view the statistics & changes between that time frame. Is this possible?
Moz Pro | | SavvyPanda
Thanks!0 -
About the rankings report in the Pro Dashboard, does it track the ranking of every page on a root domain, or just the home page or whichever page you set up the campaign with?
I noticed that one of the pages on my root domain has a #5 rank for a keyword, yet the ranking report says that there are no results in the top 50. So I am assuming it is only tracking the home page. That is one thing I liked about the Rank Tracker, that it would find any page that was ranking on a root domain. Thanks, Lara
Moz Pro | | larahill0 -
IP Location at a glance - Moz toolbar
I am using the latest version of the moz toolbar in my firefox browser. The IP location feature is not working for me as demonstrated in the article. "You will now notice a new button with a flag that shows up in the toolbar as you surf the web. This new toolbar addition shows the country where this site is hosted. Click on the flag to see more details about the location and IP address. If you want to learn more, click on the IP to access WhoIs information." My flag icon is depressed and looks like a US flag, even when I browse European sites. Also, I never see the location nor IP address of other sites. Am I missing a step or is the tool not working correctly?
Moz Pro | | RyanKent0 -
Multi-languae site anda Campaigns
Hi, I need to optimize a multi-language site. It's an hotel chain website and has 4 languages. Each language version of the site must be optimize for a diferente Google Engine. The english version of the web must be optimized for Google United States, an so on. Do I need to create a new Campaign for each language? or can I use more than 4 Engines in one campaign. Thanks,
Moz Pro | | Dragut-Comunicacion0