Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to [email protected], I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What a Brand should do first for Their Blog Keyword Research or Topic Research?
Hello Moz Community, Need your help guys. What a Brand should do first for their Blog? Keyword Research or Topic Research? Like for the Brand Product page/ eCom Pages, it's quite easy > 1. Buy "Product" Online
Competitive Research | | Max_
2. Buy "Product" "Geo."
3. Product Reviews [for Guest Posting or Reviews]
4. Best "Product" Online and so But, for the blog, it's tough for searching out the keywords that will work if the SEO team or the keyword research team is not aware or don't have the detailed knowledge about the niche they are working on. Please verify: Is it a correct method that an editor should provide a "topic" to the SEO team to find relevant keywords by using a Keyword Research tool that there are enough searches for it which can provide benefits to the blog via the search engine.0 -
How to combat competitor buying who is buying links and winning.
I have a competitor who is buying links in the Featured Advertiser Links section of Washington Post online. His backlink profile is a very low # of referring domains. (38ish) with a good bit being repeats of Wahington Post. My question, is their strategy within Google guidelines, and what can I do to combat it? Thanks.
Competitive Research | | bozzie3110 -
Competitor research: No data / results displaying on Keyword tools, Aexa
Hi there! I'm trying to research a few competitors using various Keyword tools (SEM Rush, Compete, Keyword Spy -- even Alexa for high level insight). While the bulk of the competitors generate expected results through these tools (a smattering of their top organic and paid search keywords, some traffic estimations through Alexa), ONE of these competitors lists "No results" across all categories and all tools: http://www.bgstar.ca Despite this, we know that they invest heavily in search -- and my SEM Rush toolbar indicates that they have a Google PR of 5 (though I recognize that that should be taken with a grain of salt). So I'm stumped! Has anyone encountered this before? Is there something structural that they might be doing, that's blocking not only Google-based platforms, but Alexa too? Thanks for your help!
Competitive Research | | MACJ0 -
Problem Adding a competitor
I am having a problem adding a competitor at one of my campaigns... I am not sure if I should add the link of my competitor so I wont 🙂 the link is with a /path competitorlink.gr/magazine Anyone knows what should I do?
Competitive Research | | Angelos_Savvaidis0 -
Comparing competitor folder path
Hi All I've just take on a new client and need to be able to compare their performance with a competitor's sub-folder site. The competitor is http://thebestof.co.uk/local/brightonandhove/ I've tried to add that to the competition structure in my campaign but can't use "Path" info. What do you guys do in these types of cases. BTW - TheBestOf.co.uk is a franchise with a /local/CityNames across the UK
Competitive Research | | stevecounsell0 -
What if your content is way better than competitors?
Do you have any suggestions if you are being outranked by competitors even though your page has way better content and better links?
Competitive Research | | Charlessipe0 -
Competior Research & Analyse
Hi, I like to know the inbound marketing sources of my competitors, and like to know how many traffic are receiving from Organic Ranking and Adwords or Search Engine Paid Marketing. ANd it would be nice to know the social media traffic too. I am searching, but i see here compete.com, but at the moment it's too expensive for me, like to ask you if you can find me to find any software cheaper than compete. Thanks.
Competitive Research | | leadsprofi0 -
Spying On Competitors Links
Hi There, I am looking for a tool that will allow me to learn about the new links our competitors get each month. Do you know a tool that lets you know each time your competitor has a new link?
Competitive Research | | liors0