Screaming Frog - What are your "go to" tasks you use it for?
-
So, I have just purchased screaming frog because I have some specific tasks that need completing. However, looking at Screaming Frog generally, there is so much information I was wondering for those who use it, what are the top key tasks you use it for. I mean what are your "go to" things you like to check, that perhaps are not covered by the Moz Crawl reports.
Just looking for things I perhaps hadn't thought about, that this might be useful for.
-
Ha ha, I know! It's like giving the developers a little present all wrapped up with a bow...here's the problem, and here's where to fix it
-
Allie,
That's a great example use-case. After my audits, clients are like "you found thousands of internal redirects and 404s - where are they?"
I'm like - hold on I have a spreadsheet of that!
-
I love Screaming Frog! One use case I've used recently is using it to find internal 404 errors prior-to and immediately-after a major site redesign.
After running a crawl, go to Bulk Export > Response Code > Client error (4xx) Inlinks and download the report. It shows the offending URL and the URL referring to it, which makes it easier to update the bad link.
I also have this page bookmarked, and it's my go-to guide:
-
It's one of the best tools so I feel like I use it "for everything." But some includes:
-
Title / meta duplication & finding parameters on ecomm stores
-
Title length & meta desc length
-
Removing meta keywords fields
-
Finding errant pages (anything but 200, 301, 302, or 404 status code)
-
Large sitemap export (most tools do "up to 500 pages." Useless.)
-
Bulk export of external links (what ARE we linking to??)
-
Quickly opening a page in Wayback Machine or Google cache
-
Finding pages without Analytics, as was mentioned.
I use Screaming Frog for tons of other things. Finding the AJAX escaped frag URL, identifying pages with 2 titles, 2 canonicals, 2 H1 tags, etc. Even seeing www & non-www versions live, links to pages that shouldn't be linked and http vs https.
Very cool tool - useful for pretty much everything! haha
-
-
That's awesome. Thanks. Will take a look at all those things this week.
-
I use SF religiously for all the audit work I do. I run a sample crawl (using Googlebot as the crawler) to check for all the standard stuff and go further.
My standard evaluation with SF includes:
- Redirect / dead end internal linking
- Redirect / dead end "external" links that point to site assets housed on CDN servers.
- URL hierarchical structure
- Internal linking to both http and https that can reinforce duplicate content conflicts
- Page Title/H1 topical focus relevance and quality
- Confusion from improperly "nofollowing" important pages (meta robots)
- Conflicts between meta robots and canonical tags
- Slow page response times
- Bloated HTML or image file sizes
- Thin content issues (word count)
- Multiple instances of tags that should only have one instance (H1 headline tags, meta robots tags, canonical tags)
-
That crawl path report is pretty cool, and it led me to the redirect chain report, which I have a few issues to resolve with that with a few multiple redirects on some old links. Fantastic stuff.
-
I am a big fan of Screaming frog myself. Apart from the real basic stuff (checking H1, titles,...etc) it's also useful to check if all your pages contain your analytics tag and to check the size of the images on the site (these things Moz can't do).
It's also extremely useful when you're changing the url structure to check if all the redirects are properly implemented.
Sometimes you get loops in your site, especially if you use relative rather than absolute links on your site - Screaming Frog has an extremely helpful feature: just click on the url and select "crawl path report" - which generates an xls which shows the page where the problem originates
It's also very convenient that you can configure the spider to ignore robots.txt / nofollow / noindex when you are test a site in a pre-production environment. Idem for the possibility to use regex to filter some of the url's while crawling (especially useful for big sites if the they aren't using canonicals or noindex where they should use it)
rgds,
Dirk
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Thoughts on adding "near me" to title tag for local SEO?
I want to lean out my title tags and will most likely be doing an A/B test. They currently have the "Near Me" modifier in there, which I believe Google can distinguish local SEO without it. Thoughts?
On-Page Optimization | | imjonny1230 -
Using Bold text for keywords
Hello I am updating an old e-commerce website of mine and many keywords are in bold - shall I remove the bold tag or keep them there? This is for SEO.
On-Page Optimization | | xdunningx0 -
Dashes "-" in keyword?
Just running over the page/keyword analyzer and Moz picked up the fact that my link and title are not the same as the keyword I am targeting. I am targeting the keyword "Battlefield 4 CD Key" However my title (and therefore link) are Battlefield 4 CD-Key. Note the dash. Does the dashes matter in SEO or should I try to remove them and have continuity through all of the page.
On-Page Optimization | | MrPenguin0 -
How to use canonical with mobile site to main site
I am pretty sure that the mobile version of the main site needs to be the same canonical link from what I understand. I am trying to find good docuementation that supports this. Even better if its from Google or Matt Cutts. I have a main domain like http://www.mydomain.com the mobile version of this is http://www.mydomain.com/m/ Should my canonical be rel="canonical" href="http://www.mydomain.com"/> for both these pages?
On-Page Optimization | | cbielich0 -
Sitemap error is reported when using a sitemap-index generated by Yoast
I've installed the Yoast SEO Plugin for wordpress and I've setup the sitemaps using it. I saw the tool has generated the Sitemap index file http://www.phraseexpander.com/sitemap_index.xml with different indexes for posts and pages I've submitted that to google and it's indexed. When I use Seoquake to check my website, I see that it says that the sitemap is missing (in fact http://www.phraseexpander.com/sitemap.xml) is returning 404. Shall I fix that? Shall I do a 301 redirect in my .htaccess file to http://www.phraseexpander.com/sitemap_index.xml Thanks.
On-Page Optimization | | nagar0 -
Replacing "_" with "-" in url, results in new url?
We ran SEOmoz's "On-Page Optimization" tool on a url which contains the character "_". According to the tool: "Characters which are less commonly used in URLs may cause problems with accessibility, interpretation and ranking in search engines. It is considered a best practice to stick to standard URL structures to avoid potential problems." "Rewrite the URL to contain only standard characters." Therefore we will rewrite the url, replacing "_" with "-". Will search engines consider the "-" url a different one? Do we need to 301 the old url to the new one? Thanks for your help!
On-Page Optimization | | gerardoH0 -
URL best practices, use folders or not ?
Hi I have a question about URLs. Client have all URL written after domain and have only one / slash in all URLs. Is this best practice or i need to use categories,folders? Thanks
On-Page Optimization | | 77Agency0 -
Site architecture for spatial location: Countries, states, regions: How deep should I go?
Hi, Based on the answers to my question about how to put the spatial location in the URL I'm now thinking about whether and how to flatten my information architecture. My main content is trails and courses. For both categories I have most content for Vancouver, BC (over 100 trails). I have some trails from California and more trails from other areas in BC (5-20 trails for 3 separate counties). My current site architecture is: trails -> country -> state/province -> county/regional district -> list of trails. So a trail page is 5 clicks away from the root. My course structure is: courses -> course list (I have far fewer courses but need to start structuring them) I did a search for site:example.com and found that my course pages rank most highly (probably because I have more inbound links for them) then I get workout pages then I get trail pages last of all. I want to be set up to scale for the rest of the world but I think I have to start winning in my local area first. What ideas might be good for a better site architecture? I'm thinking of doing this: trails -> location page -> list of trails for county. The location page would be a single page with a tree hierarchy from country to county - nicely styled to help the user. Something like: Canada -> British Columbia -> -> Greater Vancouver -> -> Okanagan-Similikameen -> -> Squamish-Lilloet United States -> California -> -> Marin I would make the urls be /trail/ca-bc-greater-vancouver/baden-powell-trail. I'm considering whether /trails/ca-bc/ (i.e. to get the state) should return a list of the counties. I'm worried about duplicate content for doing this. Curiously, my competitors don't have this structure at all. Access to their trails is by searching. Thoughts? Many thanks in advance
On-Page Optimization | | esarge0