System of Systems

A few years ago I was making a living as a dedicated employee of a security consultancy whose name I won’t mention. For those of you who know me, I’ll give you three guesses and the last two don’t count. In any case, one day I was working at a (unnamed) client site and I noticed one of my fellow consultants running RATS, an antiquated static analysis tool for auditing source code. RATS stands for Rough Auditing Tool for Security and “rough” is a suitable description of it. Similar to another tool named flawfinder, RATS greps through source code for calls to functions that are considered unsafe. It prints out the source file and line number where the function call that is considered unsafe occurs. The person using RATS still has to review the programming language statements themselves to confirm that the alert displayed by RATS wasn’t a false positive.

So now I wonder.. Why even use such tools at all? Why not do a manual review of the code yourself since you’re going to have to look at the code anyway? I have no doubt that the tool can grep faster than the human eye, but the trained eye can pick up things that the source code scanner can’t. Furthermore, a dynamic analyzer can detect things that both the static analyzer and the human won’t see. I don’t want anyone to think that I’m trying to malign my old co-worker’s noble effort to get his job done–that’s not my intention at all. I am simply looking towards the future and wondering how these tasks will be accomplished five or even ten years from now.

Software assurance suites that take advantage of the dynamic code analysis paradigm have the ability to feature a zero false positive rating. Static analyzers tend to ask the question: “What appears be wrong with this code that I’m analyzing?” whereas dynamic analyzers phrase the question as: “Which input sets will yield unexpected and/or unintended program output?”

As a result of executing software directly (either natively or through emulation), the dynamic analysis approach to assuring software quality is much better poised to discover bugs without presenting a conundrum of false positives. A static analysis tool may appear to detect what is believed to be a critical security vulnerability. However, since the tool is not actually executing the program in question, what appears to be an insecurity can turn out to manifest itself as a safe operation. Conversely, what appears to be a safe operation can in actuality be another vulnerability.

A few weeks ago while I was on vacation in the Outer Banks of North Carolina I was browsing through the media archives for DEF CON 15 since I missed the conference this year (I did make it out to Las Vegas, but not until September for the SANS Institute’s Network Security event.) While I was paging through the PDF-formatted slides for the presentations that I missed, one in particular immediately caught my eye; it was entitled “How I Learned to Stop Fuzzing and Find More Bugs.” Essentially, the presenter (Jacob West of Fortify Software) was playing on the fact that most (if not all) publicly available fuzzing utilities exhibit severely inadequate path coverage benchmarks. Personally, I agree with that assertion. I also believe that Jacob’s claims were somewhat slanted based on his employment at a software company that offers a static analysis product. Although I have not yet seen a practical solution for it, I do believe that it is possible to attain an optimal level of path coverage while utilizing dynamic analysis techniques.

The qualm I have with static analysis is its nature by definition. It doesn’t execute the program being scrutinized. Okay, fine. Static analyzers have their place. Maybe the tester is caught in a situation where he or she doesn’t have permission to execute the program. Regardless, I feel such a predicament is rare and if the tester is capable of executing the program, then why not do so? Why not explore all avenues of possibility? In addition to code execution, dynamic analysis can reap all the benefits of static analysis as well. Static analysis is restricted to read-only access; this is what makes static code analysis an inferior approach to software assurance. Dynamic analyzers can get the best of both worlds.. They have their cake and eat it, too!

What’s in a nutshell? The kernel of course.. but you won’t get inside the kernel if you just stare at the shell..

Not just a cliche–an analogy that sums up the dynamic versus static debate.

Welcome to Security Objectives’ new web log site at WordPress! Here you will find regularly updated musings related to System of Systems Engineering (SoSE), computer programming, software assurance and testing, digital security, philosophy, systems theory, reverse engineering, and discrete mathematics among other technical topics. News and press releases concerning Security Objectives’ itself is of course also fair game. Be sure to check back often for a healthy dose of fascinating technical research.