Mayhem @ CGC
News, 17th August 2016
With the recent win in DARPA’s Cyber Grand Challenge (CGC), ForAllSecure is another step closer to its goal of building software technology tools that automatically find vulnerabilities. Their Cyber Reasoning System, Mayhem, beat all the others in DARPA’s CGC and then competed against human teams in the DEFCON Capture The Flag contest (in which it did perform impressively, though it was defeated completely by the humans). Fortunately, the machines have not risen quite enough (yet). Or have they?
Geek Gazette had the opportunity to interview Dr. David Brumley, ForAllSecure’s CEO and the director of CMU’s cybersecurity lab, aptly named CyLab. Here are some excerpts from the same:
GG: What got you started in the field of information security?
DB: I did my first “hack” as a phone tech support for a cable company. They had a billing system, and all new customers were supposed to be activated in it first. Problem was it went down often. Some of the tech people introduced me to a “backdoor” that allowed me to circumvent the billing system and get people up and running when the billing system was down. That got me hooked. I could control the computer; it didn’t need to control me.
After that, I put myself through college as a network admin at the University of Northern Colorado, and then was a computer security officer for Stanford, and then went back to school. Every step has added a new facet, from IT to CTF to Professor.
GG: We know your vision for security is to develop systems that automatically check the world’s software for exploitable bugs. How about FAS’s vision, and whether/how it changed after the win?
DB: The win didn’t change our vision. We saw the CGC as a great test and milestone to see if we were on the right track, and the win shows that we have some pretty effective techniques for finding vulnerabilities.
GG: What are your comments on the morality and ethics of arming a machine to hack?
DB: We think that ethics and morality are important but context matters a lot. What’s right for a business may not be right in other situations.
That being said, in our experience the main strength of the tools is to react at internet speeds, something humans can’t do. They’re not autonomous skynet evil AI that is bent on destroying the world. They are tools, and tools that allows everyone to check their software for bugs, and free people from relying only on developers.
GG: Elon Musk tweeted about the CGC, “It’s all fun & games until … https://en.wikipedia.org/wiki/Skynet_(Terminator)” . What are your comments about it?
DB: We loved that it got Elon’s attention! It would be great to check Tesla’s [AI for the same].
GG: How would the nation funded security agencies be affected by the success of Mayhem?
DB: On the one hand, Cyber is a new domain that every country is thinking about. Mayhem is a new tool, and creates new strategies and tactics just like Tanks and Airplanes did. We hope leaders are thinking about how autonomy will change cyber conflicts, and how it can be used for defense.
GG: Why are Mayhem and other CRS possible now, but were considered too difficult before? Was it changes in computation ability, or something more fundamental than that?
DB: One thing that always amazes me is how much progress you get by just hammering away at the problem. As a professor, I tend to look for intellectually new and interesting ideas, and take them to the point of showing something is possible. But I’ve found the performance and characteristics of “possible” are really different than “well engineered”. Personally one of the most gratifying things about the experience is taking research ideas from the mid-2000’s and seeing what could be done with consistent effort to make them a bit more practical.
On what’s changed, I think we’re now living in the age of automated reasoning, and it’s powered by SMT and SAT solvers. SMT and SAT were once the domain of theoreticians and hard-core formal methods people. They’ve become powerful enough that we can routinely use them to analyze software in hacking contests.
GG: Mayhem won against all the other Cyber Reasoning Systems in the CGC, but did not do as good as the other teams in the CTF. What exactly is the “human spark of creativity” that the machines lack?
DB: There are a bunch of things. First, DEFCON used a different API than CGC, and that affected Mayhems’ performance and I don’t think we ended up with a clear measurement.
Based upon anecdotal observations, I think Mayhem was superior at reacting quicker than humans for things like patch generation and reflecting exploits. Shellphish used their CRS to augment their team for reaction, and from what I heard were pretty happy with how quickly it reacted as well.
Humans are way better at filtering unnecessary information that bogs computers down. The best hackers have a huge talent for quickly discarding irrelevant facts, and CRS’s lag.
GG: How did it feel competing against your own mentor, Dawn Song, in the CGC?
DB: Dawn is huge in binary analysis. She got the MacArthur Genius award for her work, and I owe a lot to her. It was really gratifying to be playing at the same level as her, and it’s not something I’ll forget.
GG: What are your views on the mass surveillance activities by the NSA and other such intelligence agencies?
DB: No comment.
GG: What’s the craziest thing you came across, while working on CGC? How about your research in general?
DB: See LEGIT_00004. https://blog.forallsecure.com/2016/08/10/case-study-legit_00004/
Alex, the team captain for Mayhem, at one point said “even with source I don’t see how it’s exploitable”.
GG: Finally, what are your views on Donald Trump? What do you think his reaction to something like Mayhem would be?
DB: No idea.
The CGC has definitely given the right push for more research and better automation in the field of security. The battle between securing and attacking cyber systems might never come to an end, but with the help of machines, both the sides will have a much greater advantage than they had before. And who knows, maybe an unexpected winner might emerge: the machines! #riseofthemachines
Photo by DARPA CGC & ForAllSecure