Analysis of Larry Suto's comparative case study

[Update: PDF attachment download is working now] 

In October 2007, Larry Suto released a case study entitled “Analyzing the Effectiveness and Coverage of Web Application Security Scanners,” available for reading at http://www.stratdat.com/webscan.pdf.  The study compared the results of three commercial web application security scanners, including WebInspect.  There has been much discussion in the industry about this study (for a good example, see the “Coverage and a recent paper by L. Suto” thread at http://lists.immunitysec.com/pipermail/dailydave/2007-October/thread.html).  Part of the discussion focuses on Suto’s questionable methodology & conclusions relating to application coverage, and the vagueness of his results.

Since any solid science experiment should be repeatable, SPI Labs set out to re-create Suto’s study to reasonably verify his conclusions and methodology.  In doing so we discovered significant discrepancies between our results and the results reported by Suto.  Attached is our final report (Suto_review_FINAL.pdf), where we indicate the results we received when we tested the same applications.