Can I crawl a password protected domain with SEOmoz?
-
Hi everyone,
Just wondered if anybody has been able to use the SEOmoz site crawler for password protected domains? On Screaming Frog you are prompted for the username and password when you set the crawler running, however SEOmoz doesn't. It seems you can only crawl sites that are live and publicly available - can anyone confirm if this is the case?
Cheers,
M
-
Hi,
I don't think it is possible to crawl any password protectec pages. Password protected pages are usually blocked and bots are unable access those pages.
And yes it is true that only sites that are publicly and live can be crawled.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google is indexing our old domain
We changed our primary domain from vivitecsolutions.com to vivitec.net. Google is indexing our new domain, but still has our old domain indexed too. The problem is that the old site is timing out because of the https: Thought on how to make the old indexing go away or properly forward the https?
Technical SEO | | AdsposureDev0 -
Is repurposing an old sub domain better than creating a new sub domain?
We have a good sub domain like** art.ourwebsite.com** which currently sells custom canvas art. We have owned the domain since 2013 but it has only been live for the past few weeks. We want to redesign & repurpose the page to continue to sell custom canvas art but will eventually include other merchandise like mugs, tshirts, etc which wouldn't be custom. Would it be best to keep art.ourwebsite.com since is a shorter/more memorible & older sub domain or would it be best to update the name to something that encompasses our new products? Our marketing team has suggested yourart.ourwebsite.com
Technical SEO | | sb10301 -
Increase in Crawl Errors
I had a problem with a lot of crawl errors (on Google Search Console) a while back, due to the removal of a shopping cart. I thought I'd dealt with this & Google seemed to agree (see attached pic), but now they're all back with a vengeance! The crawl errors are all the old shop pages that I thought I'd made clear weren't there anymore. The sitemaps (using Yoast on Wordpress to generate these) all updated 16 Aug but the increase didn't happen till 18-20. How do I make it clear to Google that these pages are gone forever? Screen-Shot-2016-08-22-at-10.19.05.png
Technical SEO | | abisti20 -
Will Links to one Sub-Domain on a Site hurt a different Sub-Domain on the same site by affecting the Quality of the Root Domain?
Hi, I work for a SaaS company which uses two different subdomains on our site. A public for our main site (which we want to rank in SERPs for), and a secure subdomain, which is the portal for our customers to access our services (which we don't want to rank for) . Recently I realized that by using our product, our customers are creating large amounts of low quality links to our secure subdomain and I'm concerned that this might affect our public subdomain by bringing down the overall Authority of our root domain. Is this a legitimate concern? Has anyone ever worked through a similar situation? any help is appreciated!
Technical SEO | | ifbyphone0 -
Changing Domain Name
Hi all, A client has just got their .edu domain and they want to change their current domain name (a .com) to this new .edu domain. The domain's CMS is Wordpress. Please correct me if I'm wrong, but basically I will need to create a new site (but they want to keep current design), move everything across to the new domain name, and 301 URL per URL? What about all the citations that the old URLs have gotten? The website is listed on Google listings/maps for some of their local keywords. Is there anyway to preserve this? Thank you all in advance.
Technical SEO | | EdwardDennis0 -
Is any know if seomoz update for site crawl.
i belive my site www.breeze-air.com hit by penguin; i found that i had un-natural anchors text and able to remove around 1200 from the 1900 seomoz found. seomoz still shows those anchors - but when i check the link its not there. i removed them 3-4 weeks ago any idea?
Technical SEO | | eoberlender0 -
Two spelling of a domain
I have a customer with two spellings of their domain name. I set up an account for spelling A and forwarded all the email boxes to spelling B becuase people tend to remember spelling A more of the time. Spelling B is the real web site. I also want any www. traffic for spelling A to go to spelling B so I used this .htaccess file in the root of spelling A Options +Indexes +FollowSymLinks
Technical SEO | | freestone
RewriteEngine On
RewriteRule ^(.*)$ http://www.B.com/$1 [R=301,L] I use to just forward A to B from the registrar but made this change to allow for email spelled either way. My question is does this create a duplicate site issue for the bots? Is this in anyway an SEO negative and if so is there a better way to do this. Thanks jw0 -
How do you store your passwords?
I have seen many solutions and tried most of them. But when serving multiple companies the amount of passwords one has gets astounding. It is hard to find a way to securely store them, but also have the convienence of looking them up. What solutions do you recommend?
Technical SEO | | MarloSchneider0