A question I’ve gotten occasionally is some version of “If you were starting over, what would you do differently with FAIR?” With the FAIR Institute now standing on its own as an independent organization, it feels like a good opportunity to answer that question.
First, however, don’t think of this as a retraction or a victory lap. Instead, think of it as a check-up. The cyber risk landscape has changed a lot over the past twenty-plus years, and I’ve learned a lot (or at least I’d like to think I have).
There have been some important adjustments in the model since my first white paper on FAIR was published in 2005, but by and large it’s held up remarkably well under years of use, misuse, and criticism.
What may be most satisfying is that the model has proven resilient to a changing risk landscape. It’s also been used to analyze risk in many domains other than cyber. This reflects the fact that the model describes risk, writ large, rather than something that’s cyber specific.
Beyond the model itself, the choice to insist on using ranges and distributions, rather than single-point estimates, was important. Faithfully expressing uncertainty isn’t a weakness of quantitative analysis; it’s a key criterion of good analysis. This is true whether you’re relying heavily on subject matter expert estimates or empirical data.
Likewise, emphasizing the importance of definitions has been crucial. Of course, some people find FAIR’s precision about terms a little pedantic. I get that, especially given that there’s been almost zero discipline around nomenclature in our profession. But a profession that can’t consistently apply its own terminology can’t measure anything reliably, and a lot of the bad risk analysis and risk analysis solutions I’ve seen over the years went wrong at the definitional level long before they went wrong at the math.
Lastly, the focus on loss event scenarios (i.e., “risks”) has helped organizations focus on what it is they’re trying to measure and manage in the first place.
That’s the good news. Now let’s look at some of the things I’d do differently (in no particular order).
While I was developing and applying FAIR as a CISO, I did so specifically to drive (and sometimes defend) decisions. That said, as FAIR began to move into more common use in the profession, I’m not sure the focus on decision support was always emphasized enough. Too often, it seemed like some of us were enamored with the act of analysis and measurement. This sometimes led to a perception of CRQ being a low-value academic exercise, rather than something that can materially improve an organization’s ability to cost-effectively apply its limited resources.
In retrospect, I’d emphasize understanding which decisions are likely to benefit most from good risk measurement and applying FAIR to support those. This would not only make the value proposition more obvious, but would also likely reduce the number of analyses organizations perform.
For a long time, the number most FAIR analyses led with was some form of annualized loss exposure. It’s convenient, it’s one number, it’s in dollars, and it fits neatly into a budget conversation. And for frequent, modest events, it works reasonably well.
However, for relatively rare, severe events, it actively misleads. Consider a scenario expected to occur about once every ten years, with a likely loss of $20 million when it does. Annualize that and you get something around $2 million a year. Now ask an executive what that $2 million tells them. It doesn’t tell them how bad a single event would be (which is the thing that keeps a CFO up at night) and it doesn’t tell them how likely the event is this year. What it describes is a loss the organization will almost never actually experience because few if any years will cost anything close to $2 million. In reality, most years will cost nothing, and the bad year will cost ten times that.
If I were starting over, the headline output for each scenario would be a per-event loss exceedance curve, answering “if this happens, how bad could it get, and with what likelihood?” Frequency would be communicated separately, as the probability of one or more events in the coming year. Annualized figures still have a place, particularly when comparing many scenarios or setting budgets, but they’d be a supporting number rather than the lead.
Yes, this asks executives to hold two ideas instead of one, but in my experience they’re perfectly capable of it. They do it every day with the many other things on their plates, so let’s give them the credit they deserve.
One of the most legitimate criticisms quantitative risk analysis faces is the question of verification: how do we know our analyses are accurate? We produce forecasts, but we rarely go back to see how they turned out.
In their book Superforecasting: The Art and Science of Prediction, Philip Tetlock and Dan Gardner emphasize that timely feedback is one of the keys to improving someone’s ability to forecast. Weather forecasters get better because every forecast gets scored the next day. Similarly, physicians get better, in part, because outcomes come back to them: the patient recovers, or doesn’t, and the diagnosis gets revisited. Most risk analyses, FAIR-based or otherwise, are filed away and never compared to reality. As a result, if an organization’s analyses have been systematically overestimating the frequency of loss events, and I suspect many have, nothing in the typical process would ever reveal it.
It’s fair to point out that cyber loss events are rare enough at the individual organization level that scoring a single forecast tells you very little. That’s true, and it’s one reason a feedback loop never got built. But we can evaluate forecasts as a population rather than one at a time. An organization with fifty analyzed scenarios over several years has enough forecasts to see whether its estimates are consistently running hot or cold.
If I were starting over, backtesting would be part of the picture from day one. And I’d frame it not as a report card that embarrasses analysts, but as the mechanism that helps us improve. I promise to publish a blog post or white paper on this soon.
The absence of a well-defined way to understand and evaluate control efficacy has long been a serious gap in our profession, which has led to a lot of sub-optimal analyses and decision-making (both inside and outside of FAIR). That gap became increasingly obvious to me over the years. Organizations would rate a control as “effective” because it existed, without asking how often it drifted out of its intended state, or how long it stayed that way before anyone noticed. For example, a firewall rule that happens to be correctly configured when an audit takes place, but is misconfigured a third of the time, isn’t the same control as one that’s right 99% of the time. But most assessments treat them identically. And behind every technical control sits a set of processes and decisions that determine its actual efficacy over time.
Of course, it’s easy to say, “I wish I’d figured out FAIR-CAM sooner,” but understanding how the control landscape works was the toughest challenge I’ve ever undertaken, and I guess things like that take as long as they take. That said, if I’d chosen to focus on it sooner, maybe we’d be further down the adoption path.
When I first developed FAIR we had to rely heavily on subject matter expert estimates for our input values. There simply was very little available empirical data. In this context, BetaPERT distributions made perfect sense. Since then, we’ve been “blessed” with many more breaches, providing data that enables us to use a broader set of statistical distributions. Unfortunately, many in the profession have come to believe that FAIR dictates the use of BetaPERT distributions for its input values. This couldn’t be further from the truth. FAIR is simply a model, and users are free to use whatever distributions best fit their input data. In retrospect, this fact should have been published much earlier.
Knowing what terms like TEF and Susceptibility mean is necessary, but it isn’t sufficient. I’ve reviewed analyses by people who clearly knew the terminology and still performed analyses that were structurally broken: frequencies multiplied twice, the same loss counted in two places, and ordinal scores fed into math as if they were measurements. The terminology was right, but the reasoning wasn’t. If I could go back, I’d put far more emphasis on the failure modes analysts fall into, and on having people build and diagnose analyses, not just define the parts.
The good news is that the training being developed at the FAIR Institute is going to correct this.
A common concern I hear is skepticism about analysis results. And to be fair, although I have seen many very high-quality analyses, I also have seen some truly awful analyses over the years. The reasons for this vary widely, but a key contributing factor is that, as well-defined as the FAIR model is, there needs to be more and better guidance on its application: specific criteria for what “good practice” looks like. Ideally, this would be something auditors and regulators can use to reliably evaluate and opine on an organization’s practices in the same way they do for accounting practices.
This, too, is something the FAIR Institute has begun to work on, and I’m very excited about the implications for the profession.
Early in FAIR’s history, I warned that FAIR should never become dogma. It must evolve.
So, consider this an invitation. If you’ve been using FAIR for any length of time, you’ve probably bumped into something you’d change. We at the FAIR Institute would like to hear what it is because some of the best improvements to this model will come from thoughtful practitioners who notice things and ask hard questions. Contact Us.
Join Nick Sanna and me on November 10 at FAIR Forward, a free virtual event for FAIR Institute members where we’ll kick off FAIR Institute 2.0 and unveil the Institute’s new independent, member-governed structure, along with the working groups and standards that will shape how FAIR evolves from here. Register here.