Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to [email protected] and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my moz domain authority dropped over night
Dear Moz team Please do let us know why my domain authority (loanonmind.com) suddenly dropped over night
Moz Pro | | experts90 -
Is it possible to block Moz from crawling sites?
Hi, is it possible to stop Moz from crawling a site at the server level? Not that I am looking to do this or anything, but here's why I'm asking. I have been crawling a site that is managed (currently by 2 parties), and I noticed that this week pages crawled went from 80 (last week) to 1 page!! I know, what? See my image attached... and the issues all went to zero "0"....! So is it possible that someone can't prevent Moz from crawling the site at the server level? I checked the robots.txt file on the site, but nothing there. I'm curious. dYNUwjd.jpg
Moz Pro | | co.mc0 -
Campaign Keyword Overview?
This may be more of a developmental comment, however... Is there any way to pull an overview of keyword ranking over the week (or any period of time) for one of your campaigns? It looks like you can pull graphs on individual keyword performance, however not for the full set of keywords for a campaign. Anyone have suggestions on how to do this within Pro Moz... or is it best to compile the keyword data every week and track it in Numbers or Excel? Thanks everyone!
Moz Pro | | Treefrog_SEO0 -
SEOmoz campaign - keyword rankings...
Hi there, I have a quick question: Does it make a difference if I add keywords to a campaign with the '[' and ']' brackets included? I just realized that I had copy/pasted many of my keywords from the exported CSV from the Google AdWords Keyword Tool and I fear that the keyword rankings on SEOmoz might factor the brackets as part of the keywords themselves. Thanks in advance! Ben
Moz Pro | | BenHammer0 -
How do we use SEOmoz to track Local Searches? Is there a way to set the location from which the campaign tracker is "searching"?
It seems that there is no way to set a parameter for location. In Places, I'm able to define my targeted region. How does SEOmoz mimic that localization? Thank for any help! -- Chris
Moz Pro | | ChrisPalle1 -
Drop in number of Pages crawled by Moz crawler
What would cause a sudden drop in the number of pages crawled/accessed by the Moz crawler? The site has about 600 pages of content. We have multiple campaigns set up in our Pro account to track different keyword campaigns- but all for the same domain. Some show 600+ pages accessed, while others only access 7 pages for the same domain. What could be causing these issues?
Moz Pro | | AllaO0 -
SEOmoz pro campaign keyword rankings.
Using the tools in the SEOmoz pro repertoire I have inserted keywords. Will the rankings tool show me the rankings for any of my pages on the site that have listed for this keyword, or just the home page?
Moz Pro | | CompleteOffice1