Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users
-
Hello!
I am looking for a spidering tool that:
- Is Mac-friendly
- Can render the DOM and find JS links
- Can spider password-protected sites (prompts for password and then continues spider, etc.)
- Has competitive pricing for 8+ users.
Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there. But if anyone has any other ideas, I've love to hear them. Thanks!
-
So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.
I was impressed, however, with the incredibly helpful team from Deep Crawl. This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory. Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into. Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.
-
Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.
So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.
-
Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.
Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.
-
Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!
-
Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Onsite search engines that are SEO friendly - which do you recommend
Hi - I am seeking an onsite search engine that is SEO friendly - which do you recommend? And has anyone tried doofinder.com - that specific search engine - if you have, is it well aligned/attuned to the SEO aspects of your site? Thanks as ever, Luke
Intermediate & Advanced SEO | | McTaggart1 -
Multiple H2 tags
Is it advisable to use only one H2 tag? The template designs for some reason is ended up with multiple H2 tags, I realise if any think it's that each one is that are important and it is all relative. Just trying to assess if it's worth the time and effort to rehash the template. Has anyone done any testing or got any experience? Thanks
Intermediate & Advanced SEO | | seoman101 -
SEO for multiple languages [Arabic]
Hello all, I am currently managing a Marketplace that comes in two different languages: English & Arabic. The English website is, fortunately, doing quite well in terms of SEO performances but, not the Arabic one. The website has two kinds of content: Static content: controlled by me. It includes menu items, navigation, static pages etc which is properly translated among the two languages User-uploaded content: It includes ads/news posted by the user which may not be translated to Arabic if they chose not to do it. Now if somebody goes to the Arabic website and check a news item that doesn't have an Arabic translation, it will show the English title. I am assuming, serving content in a different language that is specified in the hreflang is a straight no, right?
Intermediate & Advanced SEO | | MozammilStorat0 -
Handling Multiple Domain 301 Redirects on Single htaccess file
Hello, I have a client that currently that has 9 different markets with different sub-domains on one server (aka one htaccess file.). All the sites have very similar Navigation and some of them contain the same products aka same URLs. The site is using Magento CMS and I'm trying to figure out how to redirect some products that have been removed from one of the stores. The problem I'm running into is when I try to redirect one store url, it redirects all the site's URLs. Example http://store.domain1.com/ http://store.domain2.com/ I'd like to redirect http://store.domain1.com/old-url.html to http://store.domain1.com/new-url.html without making http://store.domain2.com/old-url.html redirect. I've literally been pulling out my hair trying to figure this one out but have had no luck. Does anybody have any ideas on how I could do this without having the sites redirect or create any loops? Any wisdom from you apache experts would be greatly appreciated. Thanks, Erik
Intermediate & Advanced SEO | | Erik-M0 -
Can I Use Multiple rel="alternate" Tags on Multiple Domains With the Same Language?
Hoping someone can answer this for me, as I have spent a ton of time researching with no luck... Is there anything misleading/wrong with using multiple rel="alternate" tags on a single webpage to reference multiple alternate versions? We currently use this tag to specify a mobile-equivalent page (mobile site served on an m. domain), but would like to expand so that we can cover another domain for desktop (possibly mobile in the future). In essence: MAIN DOMAIN would get The "Other Domain" would then use Canonical to point back to the main site. To clarify, this implementation idea is for an e-commerce site that maintains the same product line across 2 domains. One is homogeneous with furniture & home decor, which is a sub-set of products on our "main" domain that includes lighting, furniture & home decor. Any feedback or guidance is greatly appreciated! Thanks!
Intermediate & Advanced SEO | | LampsPlus0 -
Why would a domain show up in Webmaster Tools but not latest links?
I am going through links and trying to figure out what to disavow. I found a domain under "Who links the Most" I wanted to see what the exact link was and I can't find it when I download all links. Why would that be?
Intermediate & Advanced SEO | | EcommerceSite0 -
301 redirect to multiple domain
Hi guys, I have a domain A, B and C. The domain A was an association of two business and they are about to split. Parts of domain A are going to be redirect to domain B, but some content belong to the domain C. So my question : Is it possible to 301 redirect some pages from A to B and some other pages from A to C and if yes, what would be the impact on SEO ? Thanks a lot!
Intermediate & Advanced SEO | | StevePatenaude0 -
Ok to use rich snippets for same product on multiple pages?
I am developing a new set of pages for a series of products which exist on separate sub domains linked to the root domain. The product pages on the sub domains have rich snippets; review count, review score etc. The new pages im building out are for the same products though on the root domain and with different content. Im not comfortable marking those pages up with rich snippets too given they will have the same review counts, scores etc though would like to if its viable? Any thoughts/opinions? Thanks, Andy
Intermediate & Advanced SEO | | AndyMacLean0