How do get Moz to spider a Development site PRE LAUNCH?
-
Hi,
Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it?
Thanks
-
That would do it! Try this in robots.txt:
User-agent: Rogerbot
Disallow:User-agent: *
Disallow: /Keep in mind that this will only block crawlers that obey robots.txt. Once Roger crawls your site, it may be a good idea to lock the dev site down entirely.
-
Well, you could block Google and Bing from crawling your website (via Robots.txt) and run a Crawl via Moz, that's possible I think.
Istvan
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz unable to crawl my Zenfolio website
Hey guys, I am attempting to optimize a website for my wife's business but Moz is unable to crawl it. Zenfolio is the web hosting service (she is a photographer). The error message is: **Moz was unable to crawl your site on Apr 1, 2019. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide. I did read the troubleshooting guide but nothing worked. My robots.txt file disallows a few bots, but not roger bot. Anyone have any idea what is going on? Or do I need to request server logs from Zenfolio? Thanks
Getting Started | | bpenn111 -
Getting Started: Multiple Campaigns VS Single Campaign
I'm new to Moz and am trying to determine whether to create multiple campaigns broken out into product categories or a single campaign and label keywords by product category. I figure the majority of the folks in the Moz Community are in agency space, so most probably prefer a single campaign per client. We're an in-house marketing team so we have the ability to track up to 10 campaigns but only need to track a single website. I like the idea of having the different campaigns for different product categories, but I don't want to make things overly complex either. Just because you can do something doesn't necessarily mean you should.
Getting Started | | DFrey1 -
Is Moz Pro the tool that SEO agencies are using to SEO my website?
I was just wondering if MOZ PRO is the tool that SEO agencies and specialists are using to SEO my website? Is it possible for a new guy in the SEO to optimize his web site by himself using MOZ PRO and rank on the top? What is the best material to learn from outside the MOZ.com sources? Thank you! Kaloyan
Getting Started | | Prabuinc1 -
I haven't used Moz and you've been billing me. Please refund thanks
I signed on for a free trial and found out today that you've billed me in June and July. Check my history as this is only the second time i've logged in and it's just to request you refund.
Getting Started | | Completebanker0 -
Getting started with moz
Hi i use to be a user of SEO moz before the change to just Moz, however i am struggling to navigate around and work on campaigns. I need to know what steps to take methodically from setting up a campaign, keyword research, competitors and monitoring SERPs results. in addition improving the page grade reports and analysing keyword difficulty. Whilst Moz assist you to set up some of this initially when creating the campaign i feel it doesn't seem to take you through a logical methodical step process of configuring in depth the steps of ensuring the correct settings are relevant to the campaign i.e. an outline of the steps and i need to take that follow on from each other and getting your campaign completed. for example: create campaign keyword research onpage optimisation competitor research link finder analyser run reports that related to the info provided above I feel lost in the features of moz whilst i can see they are highly beneficial putting them to use in a chronological order to ensure the the correct setup and make use of these tools. i.e. where to start and where to end currently i feel i can only find where to start and what i should do after that to make use of Moz fully is somewhat missing. Thanks in advance, any links and direct appreciated, i would alsolike to possible speak to a Moz team member regarding my account setup if possible.
Getting Started | | mari-rose0 -
Not able to access Moz pro page and my campaigns
Hi, I could not able to access my campaign pages (pro home page). I received the below issue while logged into my account. This webpage has a redirect loop The webpage at http://moz.com/pro/home has resulted in too many redirects. Clearing your cookies for this site or allowing third-party cookies may fix the problem. If not, it is possibly a server configuration issue and not a problem with your computer. Learn more about this problem. Error code: ERR_TOO_MANY_REDIRECTS I have checked on both firefox,chrome browsers after cleared the history and cookies. Also checked with different ISP providers. But received the same. Has anyone experienced the same? Plz help.
Getting Started | | HCSEO510 -
Getting errors -- Pages with Title Missing or Empty
I am getting a high priority warning that I have pages with title missing or empty, however, when I check the page, there is a title tag <title>Long Term Care Planning</title> . Anyone have any idea what could be doing on? I check the CMS (using AEM - adobe experience manager CQ5) and the title field is filled out as well.
Getting Started | | Laura-Genworth0