Dealing with the “unknown unknowns”

“There are known knowns. These are things we know that we know.

There are known unknowns. That is to say, there are things that we know we don’t know.

But there are also unknown unknowns – the ones we don’t know we don’t know.”

Former US Secretary of Defense Donald H. Rumsfeld, February 12, 2002.

This quote is often used as an example of “political double-speak”; just reading it can give you a headache, but does it actually make sense? I believe it does, particularly when you consider that the origins of the concept it describes lie in 1980s risk management theory. Once you understand that, you might even say that best practice for anti-malware and anti-spam technology has been guided by it.

Known Knowns: classic signature-based antivirus solutions require a threat to be identified then sent to a virus lab where it is disassembled and a ‘signature’ for it is created. This approach provides strong protection against identified threats (known knowns) but is vulnerable to as yet unidentified variations of them (known unknowns). Most modern viruses are now engineered to be able to evade detection by automatically altering their code enough to ensure they don’t exactly match known signatures, effectively rendering them ‘known unknowns’.

Known Unknowns: because of the limitations of signature-based detection described above, a new method was devised without the labor-intensity and time-lag of a signature-based approach. Commtouch pioneered this new method – known as behavior heuristics – bringing the world’s first heuristics-based antivirus product to market in 1991. Heuristics uses various methods to analyze either entire program code or scraps of it, comparing them against scraps from known viruses or virus-like functions. While this method is effective at picking up as yet undetected variants of known viruses (known unknowns), it is ineffective at detecting viruses that contain previously unseen code functions (unknown unknowns).

Unknown Unknowns: this area forms today’s main battleground in the continuing fight between security vendors and the malware and spam “industry”. The industry targets so-called “zero-day” system vulnerabilities. These are vulnerabilities that have not yet been identified either by the security community at large or the vendors of the application having the vulnerability, making them ‘unknown unknowns’. Developers exploiting zero-day vulnerabilities can penetrate signature and heuristics based defenses and operate unseen on computing infrastructure. The success of hackers in identifying zero-day vulnerabilities in operating systems and applications has directly led to the rise of “bot” or “zombie” networks where spammers and malware developers take over large numbers of computers to conduct their malicious activities.

Targeting the “Unknown Unknowns”

Until recently there was no real defense against these attacks other than to respond as rapidly as possible once a threat is identified. While this minimizes the continued impact of these attacks it does nothing to mitigate the impact of the attacks prior to identification, which can be substantial.

Security vendors now focus on detecting and closing down zero-day attacks, but how do you detect something never seen before? Again Commtouch is leading the way. Our Recurrent Pattern Detection (RPD) technology examines many billions of messages globally every day to identify content-agnostic patterns without analyzing the actual content of the message, preserving communication privacy. The patterns from individual messages are compared to the patterns from all other messages, looking for structural and distribution similarities which could indicate their being part of a new outbreak. Although checks are initially done locally, patterns never seen before are compared to a database located “in the Cloud” within the Commtouch global datacenter network, or GlobalView Cloud. Because RPD can distinguish between patterns representing large-scale legitimate business correspondence (such as newsletters) and those of unsolicited bulk emails or spam, false-positives (legitimate emails being stopped by mistake) are virtually non-existent, regardless of the native language of the email. While this approach is relatively new, it has already delivered great results in addressing the growing challenge of zero-day (unknown unknowns) vulnerabilities.

So what’s next?

Academic philosophers suggest there is yet a fourth category, the “unknown known”, i.e. things we pretend we don’t know, but really do, which is really “being in denial”. Perhaps this will be the next security battlefield – what do you think? Whether it is or not, the malware and spam industry will continue to look for innovative new ways of penetrating system defenses and Commtouch will continue to lead in innovation for the defense.