Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to [email protected], I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What tools can give reliable competitor traffic data?
Could you give details of reliable tools that can measure and provide competitors' traffic data? Google Adplanner used to provide such information until June of last year, and while it's numbers didn't match Google Analytics data, the numbers were reasonably close. The metrics we are most interested in are visits, page views and unique visitors. We have looked at sites such as Compete -- but feel it is far too inaccurate/ skewed with US data only; also, would prefer to not rely on Alexa traffic stats. Thanks in advance.
Competitive Research | | ontarget-media0 -
Competitor reporting
We are in the process of creating customer profiles, conducting in-depth keyword/competitor research, fleshing out a content strategy, addressing old content, meta data etc, and researching sites (particular industry news sites) relevant to our industry who could be sources of authoritative backlinks. Once we receive approval from the powers that be, we'll start putting our strategy in motion. However, we are looking at SEO efforts as a long-term plan the entire organization understands is necessary. If our strategy is sound, I would think we'll start seeing results and some competitors will begin to take notice (and action.) So here is my question: How easy is it for a competitor to "report" a small/medium size business as breaking Google's rules i.e. could a competitor claim we are scraping their content (or content off of any site for that matter) and anonymously email Google? I think the answer to that question is "yes." How closely would Google look into the claim? We have zero plans on doing anything shady or underhanded but I was wondering if it is a simple process for a competitor to impact our efforts this way? Thanks for any insight.
Competitive Research | | SEOSponge0 -
Problem Adding a competitor
I am having a problem adding a competitor at one of my campaigns... I am not sure if I should add the link of my competitor so I wont 🙂 the link is with a /path competitorlink.gr/magazine Anyone knows what should I do?
Competitive Research | | Angelos_Savvaidis0 -
Fast Multiple Keyword research
Does anyone know of a good tool to drop a list of keywords into and get competitive and search volume results from? Don't want to have to enter them one by one to the Google KWT.
Competitive Research | | AESEO0 -
Keyword Research - tools
Hello all, I would like to find better synonyms for my keywords, and dig deeper to bid / place strategy into place for them. I am currently using the adwords too but it only gives me closely related keyword ideas. Is there something "free" which can give me a better co-relation data to work with? Thanks Aditya
Competitive Research | | shanky10 -
Competitor Traffic
Is there any way to know what a competitors traffic is? My website seems to be continuously falling despite attempts to fix and just curious if it's us or an industry issue? Thanks in advance
Competitive Research | | Professor0 -
Your favorite/most comprehensive guide to keyword research?
what's your favorite guide to keyword research? in lieu of having a favorite, what is the most comprehensive guide you've come across?
Competitive Research | | Mozzin0 -
Link analysis competitors
Am I missing something here or does SEO Moz only support three competitor sites to do comparisons against in each campaign? If not where do I add another website to the list to compare against? Thanks in advance!
Competitive Research | | AdenBrands0