Inaccurate assumptions
Richard’s entire argument rests on two assumptions: 1) that you have to base quantitative estimates of probability on statistically significant volumes of historical data, and 2) that someone doing quantitative probability estimates wouldn’t include forward-looking factors like potential changes in the threat landscape. As someone who has been doing quantitative risk analysis of infosec scenarios for over ten years, I can tell you that neither of those assumptions is accurate.
But let’s set aside those assumptions for the moment. In a risk analysis scenario Richard described, he said two things that bear examining:
With all due respect to Richard, the first statement appears to reflect a limited understanding of well-established methods like calibrated estimates, PERT distributions, and Monte Carlo functions. With this in mind, I’d encourage people to read Douglas Hubbard’s book, How to Measure Anything. Another useful resource would be any good text on Bayesian analysis. Regarding his second statement, those are exactly the same considerations a subject matter expert (SME) would include in forming a calibrated quantitative probability estimate. I can’t count the number of times I’ve facilitated SME’s in making exactly these sorts of estimates, often on scenarios just as difficult as the one Richard described. And when dealing with high levels of uncertainty, you reflect that uncertainty by using wider, flatter distributions, which I'll touch on again in a bit.
No dependency required?
Regarding Richard’s statement that my approach assumes (actually, requires) a dependency between the horizontal and vertical axis of the G5 matrix — why yes, it does. His doesn’t? Then what’s the purpose of the matrix? In any matrix of this sort, if you look up a value in one axis, and then the other, and converge those values in the matrix to arrive at another value, you are invariably assuming that a relationship exists. The only way his approach makes sense is if the likelihood scales are different and/or nonlinear — e.g., Moderate Likelihood means something different depending on whether you’re talking about attack likelihood or overall likelihood. If that’s the case, then where/how is that difference defined? How would I explain that to an inquisitive executive? In order to expose this problem, let’s walk through a simple scenario:
Using Richard’s logic (and 800-30’s Table G5) the overall likelihood of a DoS outage is High even though the likelihood of a DoS attack occurring in the first place is only Moderate. If I were an executive and someone from information security was asking for funds to improve our resistance to DoS attacks, here’s how the conversation might go (I’m leaving out the impact component to keep things simple):
Actually, if it was me sitting across the table from the infosec practitioner, I guarantee the conversation would go deeper than that. You can also count on the fact that I’d go apoplectic if someone tried to pass off an analysis where Overall Likelihood was greater than Attack Likelihood.
Through a quantitative lens
But in fairness to Richard, let’s look at that same scenario through a quantitative lens.
In my lifetime?
Another concern I mentioned in my original post, that Richard didn’t discuss, is that 800-30’s qualitative ratings lack a timeframe reference, which means I don’t know if a High likelihood means something is highly likely to happen this week or in my lifetime. Without a timeframe context, I have no idea what High/Moderate/etc. means, which means I can’t rely on or reasonably prioritize the results of these risk analyses. Interestingly, my experience has been that as soon as you put a timeframe constraint on a qualitative likelihood estimate, the person making the estimate is automatically (very often subconsciously) basing their estimate on numbers. A short series of questions like, “You say it’s Moderately likely to happen this year. Do you think that’s greater or less than 50%?” will almost always reveal this. It doesn’t take long to help them define a range they’d actually bet money on, at least if they're reasonably strong at critical thinking. And if they don't have decent critical thinking skills, then they shouldn't be analyzing risk.
Summing it up
The bottom line is that the likelihood of future events doesn’t care whether you’re using qualitative or quantitative values — or whether you’re basing the measurement on copious amounts of empirical data or gut feel — the probability of an adverse outcome cannot exceed the probability of the event that generates that outcome.
At the end of the day, risk analysis is supposed to help stakeholders make well-informed decisions. Currently, it seems to me that NIST 800-30 doesn't fully support this objective, and the defense/explanation that Richard offered doesn’t give me any reason to change that perspective. Readers, of course, are free to choose which perspective aligns best with their understanding of effective risk analysis and measurement.