Black Hat: Biases skew vulnerability reports for companies

LAS VEGAS: The security vulnerability reporting, analysis and patching landscape is being warped by a set of biases throughout the chain, according to researchers.

Steve Christey, principal Infosec engineer with MITRE, and Brian Martin of the Open Security Foundation said at Black Hat that the chain ranging from the researchers to the vendors, to the vulnerability databases that classify bugs is clouding the picture for executives and administrators.

Martin said: “People make security decisions, big ones, based on these stats. And that is depressing.”

The pair showed how a number of basic biases in human thinking can help to create a skewed picture of just how vulnerable a platform can be. For example, researchers may focus their efforts onto a single platform or a specific type of vulnerability for a short period of time, inflating the number of bug reports for one platform while flaws in others may go unreported or unnoticed due to lack of attention.

Even when the flaws are reported, differences in classification methods can help to create a bias in the way flaws are viewed. The researchers noted that common platforms, such as the Common Vulnerabilities and Exposures (CVE) system, can often classify or present issues in such a way that multiple individual flaws will be presented under a single entry and considered to be one vulnerability.

Further complicating matters can be vendor policies, which dictate how flaws are disclosed. While some vendors provide detailed security information with their patches, others provide little to no detail, often leaving privately disclosed issues completely undocumented.

The result, say the pair, is a complex system that will be nearly impossible to address with a simple formula. Rather, the researchers believe that the databases and groups that report flaws note the limitations in their methodology and help to inform administrators as to a flaw's impact on specific platforms and versions.

The public, meanwhile, is advised to take security figures with a pinch of salt. “Any time you see someone using stats to say one OS is superior to the other just walk away,” Martin advised. “No vulnerability data set out there can truly cover and answer that question.”