Avoiding the Next Heartbleed Bug
The general public is now aware of something that we network plumbers have known for years: in many respects, the Internet is robust, but security is fragile. The Heartbleed SSL problem is estimated to cost millions of dollars [1]. No one knows for sure when the next serious bug will show up and its ultimate price. This is not an acceptable situation for anyone, much less for businesses using the Internet for commerce.
An article by Robert McMillan in Wired[2] provides a great overview of the cause of the heartbleed problem – that the less-exciting, open source projects (such as SSL and TLS) are largely underfunded and staffed by volunteer software engineers:
… the OpenSSL Software Foundation, which raises money for the project’s software development, has never raised more than $1 million in a year; its developers have never all been in the same room. … In some ways, there’s a bug in the open source ecosystem.
McMillan proposes that the way to fix this “bug” is to add more oversight to the Internet’s underlying infrastructure:
We need a dedicated and well-funded engineering task force overseeing not just online encryption but many other parts of the net.
That may seem like a reasonable conclusion. However, how well has this worked? ICANN [3] an organization tasked with assuring the technical stability of one part of the Internet, the Domain Name System, has not done a good job [4] and has largely abandoned its technical responsibilities. Obviously, Heartbleed requires a better approach.
What exactly is the cause of the problem?
The promise of open source is that because it is open, lots of software engineers can see it, inspect it, comment on it, and fix bugs. However, that promise assumes that a large number of available, energetic, motivated, competent software engineers review and inspect open source code. That assumption is incorrect.
Certainly some bugs are found in open source code and they get corrected, but this is done on a “best effort” basis and there is no guarantee.
In addition, more and more inexperienced software engineers are drawn to open source. They often make changes without fully understanding the impact. Historically, senior, experienced engineers acted as team leaders or overseers of these projects, stopping the submission of unjustified, gratuitous code changes. However, this does not seem to be the case anymore. [5] Thus we have an environment of inexperienced amateurs checking in code changes and additions without fully comprehending the impact of what they are doing.
In the case of Heartbleed, the OpenSSL code base, has a specific set of challenges; very few engineers fully understand the architecture and design of the code. Very few engineers have the skill to walk through and inspect that code. Trying to rely on the “community” is just not going to work.
Proposed solutions
Some writers have proposed these solutions:
Unrealistic solutions...
Cash bounty for finding bugs [6]
Crowdsourcing [7]
However, the key point missing from these solutions is that there simply is not “a large network of potential laborers.” [8]
Since there are only about a dozen known implementations of SSL/TLS[7] with only 44 developers named for half of them (less than 8 per implementation) there may be as few as only about 100 developers worldwide who have had any direct contact with SSL/TLS internals. Probably less than 300 are likely conversant enough with the relevant RFCs to cross-check for proper implementation. [9]
Realistic solutions …
Those of us who have worked in software development and quality assurance for 20 years or more, know exactly what to do; the ideas are not new, and have been with us since 1975. [10]
Assuring well-designed, well-tested security software demands a team of qualified technical professionals. The design professionals typically must spend days going through a design review. The coding professionals must spend days going through a code review. [11] Then, quality assurance processes for unit testing, regression testing, and integration testing must be automated so that each time a new release is compiled, it is checked and verified again.
So how does this get organized and funded?
Open source projects have their time and place. There are super chic ones, like the Robot Operating System (www.ros.org). It has a great website and funding from the OSRF Foundation. At the other end of the spectrum are the mission critical, “unfashionable” open source projects; the ones involved with Internet plumbing and security. These open source projects need special treatment. The OpenSSL Software Foundation has raised less than $1 million. This is not a surprise; senior software engineers with this type of expertise are highly unlikely to be skilled fundraisers.
So what is the solution?
The community of Internet companies who rely on this technology must create a consortium or syndicate where they form a company to produce commercial quality releases of the software. The company would have a board of directors. The board would hire a president. The president would probably be a senior software engineer who would hire a small team of qualified experts who could engage in the design and code reviews required. The employees could rotate in for one year from the test teams of the Internet companies and then rotate out again. The funding source could be reasonably priced annual maintenance contracts from the most extensive users of the SSL/TLS technology. These would be banks, financial firms, Internet companies, the certificate issuing companies, and so on.
Commercial solutions
There is no guarantee that this approach will assure 100% quality, but the likelihood that problems like Heartbleed will be discovered, detected, and corrected prior to release is very high. Sometimes one simply needs to apply the tried and proven solutions for assuring software quality…
References
The “ifconfig” command was modified, resulting in massive breaking of automated scripts.
Ibid.
Botan: 1
cryptlib: 1
CyaSSL: 2
GNU TLS: 26
Open SSL: 13
PolarSSL: 1
Apple, Microsoft, NSS, and Oracle probably have two or three dozen per implementation over the course of time – i.e. closer to Open SSL and GNU TLS. Other places are probably one to three developers – so eight per implementation seems a plausible mean.Publication date of The Mythical Man-Month by Frederick P. Brooks
If you are not familiar with code reviews, some useful reading is Peer Reviews in Software: A Practical Guide by Karl E. Wiegers, and Code Complete: A Practical Handbook of Software Construction by Steve McConnell