Why Mac security product testing is harder than you think

1997 was something of a watershed for me: my first Virus Bulletin conference paper, and indeed my first presentation at an international conference. Not that many people were there to see it: it was, after all, about Macs, and even in the AV industry at that time people tended to underestimate the significance of malware on the Mac. And in fact, a great deal of the paper was about macro viruses, which rarely had Mac-specific payloads, but most of which spread perfectly happily on Macs that supported WordBasic. Making Mac users – who rarely used anti-virus – something of a Typhoid Mary in the corner of academia and medical research that I worked in at that time.

That 1997 paper on Macs and Macros – the State of the Macintosh Nation is only of historic interest now, of course: even I only have one Mac that still runs a pre-OS X version of the Operating System, and I can’t remember the last time I used it. As it happens, while Mac security has played a large part in my professional life since, my next 12 Virus Bulletin conference papers made little or no reference to Mac issues.

However, number 14, presented a couple of weeks ago at Virus Bulletin’s 23rd annual conference by myself and co-author Lysa Myers returns to two of my favourite hobbyhorses – Mac security, and security product testing. Lysa was working with Mac security specialists Intego at the time we wrote the paper, but is now a colleague at ESET, I’m delighted to say. Not only does she have a longer track record in the commercial anti-malware industry than I do, but she also has considerable experience in product testing with West Coast Labs. The paper Mac Hacking: the Way to Better Testing? is now available from the ESET Threat Center Conference Papers page by kind permission of Virus Bulletin.

So what’s the new paper about? Primarily, the difficulties that face testers when they test security products on recent versions of OS X, introduced by Apple’s own countermeasures against malware: especially XProtect.plist. While these measures do enhance end-user security, they make it harder to move away from the static testing model that mainstream Windows product testing has to a large extent evolved beyond. Here’s the abstract:

Anti-malware testing on the Windows platform remains highly controversial, even after almost two decades of regular and frequent testing using millions of malware samples. Macs have  fewer threats and there are fewer prior tests on which to base a testing methodology, so establishing sound mainstream testing is even trickier. But as both Macs and Mac malware  increase in prevalence, the importance of testing the software intended to supplement the internal security of OS X increases too.

What features and scenarios make Mac testing so much trickier? We look at the ways in which Apple’s intensive work on enhancing OS X security internally with internal detection of  known malware has actually driven testers back towards the style of static testing from which Windows testing has moved on. And in what ways might testing a Mac be easier? What can  a tester do to make testing more similar to real-world scenarios, and are there things that should reasonably be done that would make a test less realistic yet more fair and accurate?  This paper looks to examine the testing scenarios that are unique to Macs and OS X, and offers some possibilities for ways to create a test that is both relevant and fair.

There’s also an article for Infosecurity Magazine that summarizes the paper at some length: Mac Product Testing: After the (Flash) Flood.

Lysa and I will be returning to the topic in a series of blog articles here in the next few weeks, with the intention of clarifying the difficulties and suggesting ways in which Mac testing can be made fairer and more accurate, as well as the implications for other testing platforms.

David Harley CITP FBCS CISSP
Small Blue-Green World
ESET Senior Research Fellow