Aberdeen found that:
“Open source software, commonly used in many versions of Linux, Unix, and network routing equipment, is now the major source of elevated security vulnerabilities for IT buyers. Security advisories for open source and Linux software accounted for 16 out of the 29 security advisories — about one of every two advisories — published for the first 10 months of 2002 by CERT (Computer Emergency Response Team). Keeping pace with Linux and open source software are traditional Unix-based software products, which have been affected by 16 of the 29 — about half of all — advisories to date during 2002. During this same time, vulnerabilities affecting Microsoft products numbered seven, or about one in four of all advisories.”
If true, this is a stunning turnabout. It is common knowledge that Microsoft has had much bad press surrounding security of its products, such as deficiencies in IIS. Organizations with mission-critical security requirements have traditionally implemented open source products for systems exposed to the Internet, such as Unix or Linux. So, if Aberdeen’s analysis is correct, the trend has been reversed, with Microsoft’s efforts over the past year in “trustworthy computing” paying off to make Microsoft now more secure than Unix/Linux and open source in general.
However, Aberdeen's analysis is faulty. Because CERT advisories are public information, it is a simple matter to look at the raw data behind Aberdeen’s conclusions and see where Aberdeen erred. David Kelsheimer, network services practice director for Strativa, assisted me in dissecting Aberdeen’s conclusions. Based on our analysis of the 2002 CERT advisories, we can summarize the problems with Aberdeen’s study as follows:
1. Aberdeen counts CERT advisories, ignoring multiple vulnerabilities per advisory. This is like counting the number of guests arriving at a party by counting the automobiles they come in, regardless of the number of passengers in each car. For example, CA-2002-09 describes 10 separate vulnerabilities in Microsoft’s IIS, but Aberdeen counts them as one advisory. Thus, by Aberdeen’s reckoning, in the first 10 months of 2002, 16 out of the 29 security advisories are for open source/Linux, and 7 out of 29 are for Microsoft.
However, when we count based on the number of vulnerabilities within the advisories, the score is 18 for open source/Linux and 24 for Microsoft. We categorize another 34 as “other” or cross-platform vulnerabilities, because they are difficult to attribute to Microsoft or Linux/open source. (E-mail me at the address in the right column if you would like a copy of our worksheet.)
2. CERT advisories are not an adequate sample. CERT itself has said as much, in response to Aberdeen’s study. CERT’s comments were reported in an InternetWeek article, which said,
“CERT believes Aberdeen drew too much from its numbers. The organization doesn't draw any conclusions from its advisories on the vulnerability of open-source software vs. Microsoft or any other seller of proprietary applications. Instead of comparisons, the group focuses on identifying and studying security problems it considers most serious based on CERT's own metrics. That covers about 20 percent of all known vulnerabilities, said Shawn Hernan, senior member of the CERT technical staff.”
If Aberdeen were interested in a more complete sample, it could have looked at the complete database of CERT vulnerabilities, which lists over 3,000 vulnerabilities for the first ten months of 2002. The fact that it didn’t is puzzling.
3. Comparing Microsoft with “open source/Linux” is not a fair comparison. As one correspondent to SecurityFocus pointed out,
“…to take a listing of vulnerabilities from CERT (not a comprehensive list by any means!) and say that Linux is less secure because there are more open source advisories is laughable. There are more types of open source software out there, than there are software packages from Microsoft. To attribute open source flaws to Linux is like blaming Microsoft for the holes in AOL Instant Messenger.”
4. Aberdeen fails to note other problems with the use of CERT data. For example, CERT only reports vulnerabilities that are confirmed by the software developer. Because of the nature of open source, vulnerabilities tend to get reported and confirmed more transparently for open source than for closed source products, such as those of Microsoft, which has the option of not disclosing vulnerabilities which it finds and patches itself in the next release. Second, CERT issues advisories only for those vulnerabilities with the potential for the widest impact on the Internet. As one correspondent to OSOpinion pointed out,
“Since a large percentage of Internet infrastructure is based on open source software such as BIND, sendmail, and Apache, it makes sense that security flaws in these products would be considered serious, while flaws in Microsoft products may not be counted because they have much less of an impact. A security flaw in Microsoft Word may be bad, but it does not have the potential to bring down much of the Internet or compromise the integrity of millions of dollars in e-commerce.”
Aberdeen’s failure to properly use CERT data can only have two explanations: either Aberdeen researchers did not realize the shortcomings of such data, or worse, they had a conclusion they wanted to reach and searched for data to help them reach that conclusion. If Aberdeen’s research is so poor when based on public data, which can be independently verified, how can we trust its research when it develops the data itself and does not release it in its raw form?
