Exclude status codes in Screaming Frog
-
I have a very large ecommerce site I'm trying to spider using screaming frog. Problem is I keep hanging even though I have turned off the high memory safeguard under configuration.
The site has approximately 190,000 pages according to the results of a Google site: command.
- The site architecture is almost completely flat. Limiting the search by depth is a possiblity, but it will take quite a bit of manual labor as there are literally hundreds of directories one level below the root.
- There are many, many duplicate pages. I've been able to exclude some of them from being crawled using the exclude configuration parameters.
- There are thousands of redirects. I haven't been able to exclude those from the spider b/c they don't have a distinguishing character string in their URLs.
Does anyone know how to exclude files using status codes? I know that would help.
If it helps, the site is kodylighting.com.
Thanks in advance for any guidance you can provide.
-
Thanks for your help. It literally was just the fact that it had to be done before the crawl began and could not be changed during the crawl. Hopefully this is changed because sometimes during a crawl you find things you want to exclude that you may have not known of their existence before hand.
-
Are you sure it's just on Mac,have you tried on PC? Do you have any other rules in include or perhaps a conflicting rule in exclude? Try running a single exclude rule, also on another small site to test.
Also from support if failing on all fronts:
- Mac version, please make sure you have the most up to date version of the OS which will update Java.
- Please uninstall, then reinstall the spider ensuring you are using the latest version and try again.
To be sure - http://www.youtube.com/watch?v=eOQ1DC0CBNs
-
does the exclude function work on mac. i have tried every possible way to exclude folders and have not been successful while running an analysis
-
That's exactly the problem, the redirects are disbursed randomly throughout the site. Although, and the job's still running, it now appears as though there's almost a 1-2-1 correlation between pages and redirects on the site.
I also heard from Dan Sharp via Twitter. He said "You can't, as we'd have to crawl a URL to see the status code You can right click and remove after though!"
Thanks again Michael. Your thoroughness and follow through is appreciated.
-
Took another look, also looked at documentation/online and don't see any way to exclude URLs from crawl based on response codes. As I see it you would only want to exclude on name or directory as response code is likely to be random throughout a site and impede a thorough crawl.
-
Thank you Michael.
You're right. I was on a 64 bit machine running a 32 bit verson of java. I updated it and the scan has been running for more than 24 hours now without hanging. So thank you.
If anyone else knows of a way to exclude files using status codes I'd still like to learn about it. So far the scan is showing me 20,000 redirected files which I'd just as soon not inventory.
-
I don't think you can filter out on response codes.
However, first I would ensure you are running the right version of Java if you are on a 64bit machine. The 32bit version functions but you cannot increase the memory allocation which is why you could be running into problems. Take a look at http://www.screamingfrog.co.uk/seo-spider/user-guide/general/ under Memory.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Proper 301 redirect code for http to https
I see lots of suggestions on the web for forwarding http to https. I've got several existing sites that want to take advantage of the SSL boost for SEO (however slight) and I don't want to lose SEO placements in the process. I can force all pages to be viewed through the SSL - that's no problem. But for SEO reasons, do I need to do a 301 redirect line of code for every page in the site to the new "https" version? Or is there a way to catch all with one line of code that Google, etc. will recognize & honor?
Technical SEO | | wcksmith10 -
Does Index Status Goes Down If Website is Penalized by Google?
Hi Friends, Few of my friends told me that index status of a website goes down goes if it is penalized by Google. I can see my organic traffic has went down drastically after the Panda update on July 17th 2015 but index status still remains the same. So, I am bit confused. Any advice on this.
Technical SEO | | Prabhu.Sundar0 -
Canonical issues using Screaming Frog and other tools?
In the Directives tab within Screaming Frog, can anyone tell me what the difference between "canonicalised", "canonical", and "no canonical" means? They're found in the filter box. I see the data but am not sure how to interpret them. Which one of these would I check to find canonical issues within a website? Are there any other easy ways to identify canonical issues?
Technical SEO | | Flock.Media0 -
Some URLs were not accessible to Googlebot due to an HTTP status error.
Hello I'm a seo newbie and some help from the community here would be greatly appreciated. I have submitted the sitemap of my website in google webmasters tools and now I got this warning: "When we tested a sample of the URLs from your Sitemap, we found that some URLs were not accessible to Googlebot due to an HTTP status error. All accessible URLs will still be submitted." How do I fix this? What should I do? Many thanks in advance.
Technical SEO | | GoldenRanking140 -
Site Wide Text to Code Ratio Tool
Does anyone know of a free or paid tool which provides the text to code ratio for all pages on a site? Something like Screaming Frog but with all the ratios for each page. At the moment we are checking key landing pages individually.
Technical SEO | | Dave_Schulhof0 -
Why am I seeing %%name%% showing in the duplicate titles report when it shows the name correctly in the source code?
Crawl diagnostics is picking up all the Wordpress variable tags including and not limited to %%name%% instead of what is actually showing in the source code. Shouldn't it show what is rendered in the browser? I don't think these need to be fixed because they show in Google ok. Search Google for: site:blog.sandiego.org "About Aki"
Technical SEO | | SDConvis0 -
500 error codes caused by W3 Total Cache plugin?
Hello Everyone, I operate a site (http://www.nationalbankruptcyforum.com) that has been receiving 500 error codes in Webmaster Tools as of late. This morning, webmaster tools showed 129 500 crawling errors. I've included one of the URLs that contained an error message here: http://www.nationalbankruptcyforum.com/marriage-and-bankruptcy/do-my-wife-and-i-both-have-to-file-for-bankruptcy/ I've been getting these errors now for about 3 weeks and they've mostly been on obscure, strange URLs (lots of numbers etc.) however, this morning they started showing up on pages that will actually be trafficked by users. I'm really not sure where they're coming from, although I do believe it's a software issue as I've had my hosting company take a look to no avail. I have had some development work done recently and am running the W3 Total Cache plugin (my site is built on WP). I also run the Yoast SEO plugin and rely on it to publish an XML sitemap among other things. Anyone have any idea where these 500 errors originate from? Thanks, John
Technical SEO | | oconn1460