Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to [email protected] and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Site Crawler not working but on-demand crawler working
Hi, In Moz pro, when using Site crawler (or recrawl), we are seeing message site is banned. But when using on-demand crawler, it could generate report successfully. I just like to know if in both these cases, it is roberbot that is used! And kindly note, site crawler was perfectly working before. So the required setup is already in place from long time. Site crawler ban issue started appearing from nov/dec 2023. . Could you please us understand how could we possibly make site-crawler work?
Moz Pro | | gilesd
I am happy to provide more details if you need any. Thanks0 -
Moz Domain Authority 27
Hi there, I have been reading up about Domain Authority being important to rank though I have a question about it. When trying a few google searches and using a tool to see the Moz Domain Authority i notice the following. I won't mention sites or keywords, happy to send them via private message if required. A site ranks 1st position for most keywords including the main keyword that related to their industry, a highly competitive keyword. Their Moz Domain Authority 27, Est. Visits: 265,000, Pinterest shares 72, Facebook shares 191 and unique domains that point to it are 55. Yet when I look at another site, optimising for the same keyword, Moz Domain Authority 50, Est. Visits: 60,000, Pinterest shares 35000, Facebook shares 20000 and unique domains that point to it are 521. I would have thought that the second site ranks 1st for the keyword and the first one mentioned which is in 1st position I would have expected to be on page two or three. as I scroll down the first page for the term, all sites except for one have higher DA, visits, shares and so on. If a high PA is required can someone help me to understand the above example? Whn a low PA site ranks much better for a primary keyword than all other sites that have a higher PA than it?
Moz Pro | | IvanaDaulay1 -
Website blocked by Robots.txt in OSE
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned. Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
Moz Pro | | ConnellyPartners0 -
Once I start a campaign, is it possible to edit the domain?
I accidentally submitted a subdomain instead of the root domain. Is it possible to fix this? Or how should this be addressed? Thanks in advance!
Moz Pro | | comerecommended0 -
How well is your Internet marketing working for your business?
Not asking about specifics of money, but this is just a general question to see what kind of return people get for their Internet marketing time investment.. I would say I'm getting over 40,000 hits per year for one business which is holiday cottages but I only own one cottage and act as an agent for three other clients. Because the campaign is working well for me at the moment I get probably 95% occupancy throughout the year, which you would have no chance of accomplishing without SEO intensive input. If I had to pay £0.50 a click approximately obviously it wouldn't be worthwhile, especially when you realise that a percentage of the traffic and not related to your product. I also have a second business which is installing wireless security alarms over the Internet and once again approximate traffic at the moment is almost approaching 40,000 again per year. Now I'm sure there are some big hitters out but my general question is how productive, is you're organic results and Internet marketing for your personal business in comparison to what it would cost you if you had to pay per click? I'm kind of wondering how well the Internet is working people on this forum for their particular businesses? Good, Bad ? whats your story ? Cheers alan
Moz Pro | | whitbycottages0 -
Why does SEOMoz crawler ignore robots.txt?
The SEOMoz crawler ignores robots.txt It also "indexes" pages marked as noindex. That means it is filling up the reports with things that don't matter. Is there any way to stop it doing that?
Moz Pro | | loopyal0 -
Frequency of the Campaign reports
Hello there, I would like to know the frequency of the New Rankings and On-page reports for the campaigns that we set up. We received our last campaign report on 3rd September for all the 5 campaigns. We also received 2 reports again on 9th September. Can you please let me know how often do we get these reports and can we manage to receive them every week or fortnight. Best Regards Sandeep
Moz Pro | | CalistonDigital0 -
Tracking Local Keywords in Campaigns.
When selecting keywords to track in campaign mode, how should I enter the keywords for a company located in a city that shares the same name as another city in a different state? My assumption is that if you're searching for "Garden City Plumber" within Garden City Michigan that Google will know your IP and show local results (as opposed to results from Garden City California). If I want to track the key phrase "Garden City Plumber" in campaign mode how will SEOMoz know which city to deliver results for? Thanks in advance.
Moz Pro | | OptioPublishing
Jason0