How do get Moz to spider a Development site PRE LAUNCH?
-
Hi,
Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it?
Thanks
-
That would do it! Try this in robots.txt:
User-agent: Rogerbot
Disallow:User-agent: *
Disallow: /Keep in mind that this will only block crawlers that obey robots.txt. Once Roger crawls your site, it may be a good idea to lock the dev site down entirely.
-
Well, you could block Google and Bing from crawling your website (via Robots.txt) and run a Crawl via Moz, that's possible I think.
Istvan
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is analytics.moz.com showing up a traffic source in Google Analytics?
Under Traffic Sources, analytics.moz.com is listed as a source for 37 sessions this past month. What might this be? Is it legitimate traffic or should I be filtering this?
Getting Started | | mphdavidson0 -
Getting started with moz
Hi i use to be a user of SEO moz before the change to just Moz, however i am struggling to navigate around and work on campaigns. I need to know what steps to take methodically from setting up a campaign, keyword research, competitors and monitoring SERPs results. in addition improving the page grade reports and analysing keyword difficulty. Whilst Moz assist you to set up some of this initially when creating the campaign i feel it doesn't seem to take you through a logical methodical step process of configuring in depth the steps of ensuring the correct settings are relevant to the campaign i.e. an outline of the steps and i need to take that follow on from each other and getting your campaign completed. for example: create campaign keyword research onpage optimisation competitor research link finder analyser run reports that related to the info provided above I feel lost in the features of moz whilst i can see they are highly beneficial putting them to use in a chronological order to ensure the the correct setup and make use of these tools. i.e. where to start and where to end currently i feel i can only find where to start and what i should do after that to make use of Moz fully is somewhat missing. Thanks in advance, any links and direct appreciated, i would alsolike to possible speak to a Moz team member regarding my account setup if possible.
Getting Started | | mari-rose0 -
Why can't I Ctrl + click on links on Moz any more?
I'm interested if it's just me that gets frustrated by this? I've just Ctrl + clicked a few links to open them in separate tabs and then realised that none of them had opened. I know it's been like this for a while. It's a usability issue as it goes against expected norms, and now I have to right-click and then click "Open in new tab" on each link, which is more time-consuming and frustrating. More and more websites seem to be losing their Ctrl + click on links ability (JavaScript often breaks it). I don't know if there's a Mac equivalent... Anyway, I hope that doesn't seem like I'm too angry. It just frustrates me a little and I hope it gets fixed. 🙂 Edit - I've just realised these are getting blocked by Chrome's pop-up blocker - but why? It's only an issue on a small number of websites.
Getting Started | | Alex-Harford1 -
Why does Moz Analytics only let you look at last month / last week?
I cant think of any other reporting product that only lets you look at the last period, am I missing a setting somewhere. I need to take a look at Feb!
Getting Started | | locusm1 -
Site traffic vs other sites
I have a quick question. Does Moz have a tool I can use to see how much traffic my site gets a month versus some competition? I know Alexa has this but you have to create an account. I am just wondering if Moz has one and I am missing it.
Getting Started | | trumpfinc0 -
Fastest way to set up Moz to produce a list of easy-to-rank for keyword opps
I've used Hubspot's keyword tool previous to MOZ, and I know that they import data generated from MOZ into their tool... but I find it a much more convoluted process using Moz's more robust system. ALL I want is to dump in a long list of keyword possibilities, then see where we can easily rank for these terms, especially against our local competitors. How do I set this up???? It's not as self-explanitory as Hubspot! Thanks!!!
Getting Started | | uptongirl340 -
Google Analytics data not loading into Moz
I use several analytics apps on my iphone which require you to connect to your GA account. Each one gets immediate access to the GA data. With Moz, I connected to my GA account 3 days ago after many failed attempts due to some bug, but still have no data showing. Is there some reason why Moz would take several days to access the data where iphone apps take seconds? I am now a week into my 30 day free trial, and have spent lots of time trying to get it to work, but have not been able to do much. I'm afraid of how much of the 30 days will be taken up just getting Moz set up and working, rather than actually using and evaluating what it does. Any possibility of re-starting the 30 days from when it's actually working? I don't mind waiting patiently for some bugs to be fixed, just would like to know that I will still get a 30 day trial. Thanks.
Getting Started | | RLodato1