Vegas, baby! Iron Chef Black Hat

Draft posted August 14. Substantially revised August 17.

The second of a two-part series on the Black Hat USA 2008 security conference.

Image of Caeser's Palace from Black Hat site

Back when we lived in San Francisco in the 1990s, we were huge fans of Fuji TV’s Iron Chef, then shown with subtitles on a local cable station. When local chef Ron Siegel repeated his winning “lobster confront” menu at Charles Nob Hill, word got leaked to the Iron Chef mailing list and we managed to get seats … wow! And I’ll never forget the time that Bobby Flay in his exuberance jumped on the sushi board; so of course when I was at Caesar’s I had to have lunch at his Mesa Grill.

Iron Chef is also a good lens to looking at Black Hat from the perspective of the consulting I’m doing for San Francisco-based startup Coverity. This gives a completely different picture of the conference than the political and front-page-news of Vegas Baby! Black Hat, glitter, and pwnies. It’s just as interesting though, thanks in no small part to Fortify’s Iron Chef Black Hat.

For those not familiar with the esoteric space of static analysis tools, Coverity and Fortify are two of the major players. Coverity is strong in reliabiilty and the embedded segment; Fortify is strong in security and in the financial segment. There are other important players as well, including Ounce, Armorize, Veracode, Klocwork and Microsoft (whose PREfix and PREfast, both originally architected by me, still set the bar in a lot of people’s minds). Fortify and Coverity are natural competitors: both Bay Area startups on a path to go public, with very different styles and strengths.

I’ve been consulting for Coverity investigating the possibility of breaking into the security space. Fortify is the clear market leader in “static analysis for security” but their support in the developer community seems very tenuous. Iron Chef Black Hat is pitched at the penetration tester community, where Fortify has quite a few partners. From a competitive standpoint, what can we learn?

Robert Vamosi’s pre-conference Security Bites podcast on CNet lays it out nicely:

Brian Chess, chief scientist at Fortify Software, and Jacob West, who manages Fortify Software’s Security Research Group, tell CNET’s Robert Vamosi that one team will use static analysis while the other will use fuzzing. Chess confirmed that Charlie Miller and Jacob Honoroff will be on the fuzzing team, and Sean Fay and Geoff Morrison from Fortify will make up the static analysis team.

cover of If there were a “fantasy league” for for security competitions, Charlie and Jake would be ultra-hot commodities: along with their Independent Security Evaluators colleague Mark Daniel, they won Tipping Point’s pwn2own contest in April. On the other hand, in this contest they were using open-source tools and going up against a million-dollar commercial tool set — run by the experts. With reputation at stake on both sides, the pressure was intense ..

Tim Wilson’s Fuzzing Nips Static Analysis in ‘Iron Chef’ Contest on Dark Reading sums it up:

A panel of three security experts acted as the judges, and voted two to one for the fuzzing team. “I’m amazed at how well the static analysis team did,” said Mozilla’s Window Snyder, who cast the deciding vote. “But the fuzzing team just did a better job.”

Indeed. Amazingly, both teams found exploitable vulnerabilities within an hour. The static analysis team found a remotely-exploitable vulnerability that required a server admin to upload a MP3. This would probably be difficult to exploit in practice without some significant social engineering, but still, not bad at all for 60 minutes.

The fuzzing team, by contrast, found a different vulnerability, one that was exploitable without any user action — the most dangerous kind. They used a variety of tools for this, including Sulley and GPF (both of which found the vulnerability), ProxyFuzz, as well as valgrind and a homebrew tool for monitoring. Of course there are some excellent commercial tools out there as well; it’s still remarkably impressive how good the open-source offerings are in this area.

Window goes into more detail in Michael Desmond’s article in Redmond Developer News.

“Fuzzing has been very successful for us and found lots of vulnerabilities. My experience with static analysis has been that there are so many false positives that it can be difficult to get any real value out of it. I was impressed that these guys were able to identify what appears to be a significant issue in such a short period of time using static analysis tools, and it made me reconsider whether it was time to take another look at these tools.”

Window is one of the few people who consistently gets me to look at static analysis in a different way, so when she says stuff like this, I listen. With fuzzing when you get a hit, you know you’ve got a vulnerability that’s exploitable at least for a denial-of-service and you’ve also got the information you need to explore the exploitability. On the other hand, fuzzing’s limited by test cases, so static analysis potentially provides a valuable complement. Static analysis typically has just a tiny percentage of warnings corresponding to real vulnerabilities: a lot are false positives, a lot of the remaining ones are defects but not vulnerabilities, and a lot of the vulnerabilities aren’t exploitable. At Iron Chef Black Hat, the experts were able to cut through the noise and zero in on a real vulnerability very quickly.

Of course, people using tools they’ve helped develop (as both teams did here) unsurprisingly get results that are a lot better than anybody else does.* Still, it’s an indication that with the right expertise, these tools are powerful enough to be very useful — it’s now a matter of making them more broadly accessible, to real people not just static analysis whizzes. And an important point not to overlook: if Fortify can do this, then most likely that’s the state of the art more broadly. So it seems to me that Window’s ahead of the curve as usual, and a lot of people will be taking another look at static analysis tools as complements to fuzzing.

iron chefs in the stadium

The seven original Iron Chefs — and Kaga

Iron Chef Black Hat is a fantastic idea, tapping into the same competitiveness as pwn2own and the Capture/Defend the flag contests, and so it’s great to see it become an annual tradition. Black Hat and Fortify still have a ways to go if they want to get to the high bar that CanSecWest and Tipping Point set with pwn2own. First of all, it’s a real wasted opportunity that there’s nothing on Fortify’s blog — it would be great to know more about what defects the teams found, how they chose to focus on those sections of the code, etc. This in turn leads to a lack of blog discussion or press attention to this: I couldn’t find any coverage other than Tim’s and Michael’s.

And there’s a subtle point about the way Fortify presents the event as being basically about them, not about the participants. The Redmond Developer News article is an interview with Brian Chess of Fortify that doesn’t even mention the winners; the Dark Reading post originally misdescribed the winners as Fortify employees.** Compare and contrast with the way Tipping Point prominently featured the winners of pwn2own.

Despite these imperfections, there’s clearly a huge amount of value as well as fun in Iron Chef Black Hat. Still, I would be remiss in my duties as a strategist if I didn’t highlight some of the key takeaways for Coverity — or Armorize, Klocwork, Ounce, Veracode, anybody else competing with Fortify.

Even with the huge amount of press attention Black Hat got this year, Fortify doesn’t seem to have been able to capitalize on it. On top of that, the results certainly could be interpreted as “when used by experts, Fortify’s expensive static analysis tools are not as effective at finding vulnerabilities as open source fuzzing tools.” Fortify has other strengths as a company, but as market leader in the “static analysis for security” space they are looking potentially vulnerable.

Looking forward, the big battleground is likely to be for “tainting” defects in Java, PHP, .NET and JavaScript (as well as C/C++), which aren’t amenable to fuzzing in the same way that buffer overruns are. There’s clearly a lot of “upside potential” for static analysis here; when we first implemented some simple taint analysis in PREfix in 2001, we immediately an exploitable vulnerability in a Windows system-level component that (whew) hadn’t shipped yet. The key challenges are having an analysis that’s powerful enough that it finds vulnerabilites — and keeping the noise is at a low enough level that the tools are usable.

So it turns out there is a lot to learn from Iron Chef Black Hat. Props to Fortify for trying it, and to Black Hat for supporting them, and hopefully it’ll continue to improve over time. There’s huge potential here — imagine a couple of teams each for static analysis and fuzzing, using different tools; or showdowns in Java, .NET, and PHP as well as C/C++.

And if that’s too much to fit into an hour-long panel, hey, maybe Fortify can rent Mesa Grill and turn it into a party!


* My experience bears this out: I was consistently able to get good results on tools that I had written months or years before anybody else could. “It’s almost like whoever designed the tool knew just how I think!”

** the author corrected this immediately after Charlie and I pointed it out to him, and so I do see this as Fortify’s responsibility to a large extent: what created this misimpression? Why didn’t they follow up and ask for a correction?

Thanks to the reviewers of the previous draft for the corrections and other feedback!