Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to [email protected] and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No Ranking Keywords in Moz
I am using the link explorer to search my website https://bestsportsmask.com/ in Moz. But it is showing no ranked keywords but in other tools it's showing some keywords. What could be the issue.
Moz Pro | | DanielMaier0 -
Setting up MOZ to run on Staging
Hi Moz, We would like to setup Moz to run on our Staging Server. This would be extremely valuable as it alerts us to new SEO issues/risks in a controlled and secure environment that is not exposed to production. Our internal team has recommended potentially setting up a reverse proxy server that will validate either via Moz's/Rogerbot's http header or IP and allow Moz access to our Staging environment to crawl. Is this something that we can setup with Moz? Are there other ideas to enable Moz to crawl our Staging server?
Moz Pro | | kriskunisch0 -
MOZ Not Picking Up Facebook "Likes"
For some reason MOZ is not registering the number of Facebook likes my corporate Facebook page has. Over 700 likes have accumulated, yet MOZ is showing zero. It is strange because MOZ has picked up three Facebook "shares". Does anyone have an idea how this issue can be corrected? Thanks,
Moz Pro | | Kingalan1
Alan0 -
SEO moz Report Card
I just ran some on page report cards. As I was playing around with the tool I noticed that I would get different results if I used my primary domain vs a 2nd domain. The main difference was in how the tool was counting keywords on the page. The keyword used was 'vehicle inventory' Primary domain: www.brand-state.com/inventory.htm Title = 1, URL = 0, Meta = 1, H1 = 1, H2-4 = 1 Body =1, Strong = 1, IMG Alt = 1 Total = 7 2nd domain: www.company-name-brand.com/inventory.htm Title = 1, URL = 0, Meta = 1, H1 = 1, H2-4 = 2 Body =5, Strong = 4, IMG Alt = 2 Total = 13 I can understand if the keyword was in the domain, but it's not. So I'm wondering what is going on here - any help or suggestions on what to research would be a great help. Thank you!
Moz Pro | | gormaniavt0 -
How to set the crawler or reports to ignore
I have a mobile version of a site with a URL string that disables the mobile view on smartphones (view full site), string is like this example.com/page-name.html?mobile=off I need the seomoz pro reports or crawler to ignore it because the crawler visits both versions of the site then reports them as duplicate content. Is there a setting page I haven't visited yet that will set this?
Moz Pro | | Str82u0 -
Rank tracker not working right
so i'm quite new at using seomoz so maybe this is a silly question. a couple of weeks ago i added 6 url to tack weekly and it's been working so far. since then i've been wanting to add several more but it seems that the list is limited to 6. at first i click the checkbox on track column but when that didn't work i updated each url then click "track this url" --> that didn't work either. can anyone help if maybe i'm doing something wrong? thx
Moz Pro | | BSutandio0 -
How do data feeds work in product selection
Hi – I don't know if this is an SEO matter or not, but I have data feeds to Google, Bing and thefind.com for all of my products. I have no idea if these feeds help. My guess is that they do not generate much business. For example if I type in "Brass Bookends" our products usually don't show up until the late pages. Using this as an example does anyone know how these sites select the order in which they display products? Another way to phrase this is what makes thefind.com show a competitor's brass bookends several pages before ours? FYI we have been doing these data feeds for a year on a consistent basis so being new shouldn't be a factor. Any insights will be most appreciated. Thanks. Don. www.uniquegiftsanddecor.com
Moz Pro | | uniquegifts-2778790 -
Cant understand Seo moz analysis
Did a semoz analysis of site www.1stclassdriving.co.uk and it came up with 484 pages with duplicate titles and 484 with duplicate content. Also gave me a 531st ranking for search term "Driving Lessons" (with many smaller schools with better rankings)- so I think that I may have a problem 1-Duplicate titles - I have many area pages - one for each of the post codes that we cover (some 40 odd) Each page is titled "Driving Lessons in "xxx" where xxx is the area.
Moz Pro | | Brian_Worger
Plus each Instructor has their own page which is tiled "Driving Lessons in xxx" where xxx is the more local towns/areas covered.
Is this the use of the term Driving Lessons xxx on each page the problem? The report doesnt seem to make this clear -it simply lists each page on the site 2- Duplicate content puzzles me as the content does differ although each instructor page has probably 80% content derived from a master page. Attach screenshot - hope someone can help Many thanx Brian0