I regularly read blog posts or encounter people in our profession who dismiss quantitative cyber risk measurement as “guessing”, or “nothing more than feelings” (cue the Morris Albert song). Since this is such a common concern, I thought it would be worthwhile to examine this issue of what's subjective, what's objective and what falls between.
Feelings vs. Guessing vs. Estimating
When I’m presenting to a group of people or teaching a class on risk analysis, I’ll often point to someone in the room and ask them how tall they think I am. Almost invariably, they’ll answer with something like “You look like you’re about six feet tall”. I then ask the person two additional questions:
- Would they be willing to bet $1,000 of their own money on their answer?
- Did they believe their answer regarding my height was subjective or objective in nature?
Regarding their willingness to bet — the answer has always been “No”. They simply have too much uncertainty in the value they gave me. I then often ask if they can give me a range they’d be willing to bet their life on. After a pause, the answer to that question has always been something along the lines of, “Yes, of course. You’re somewhere between one inch and ten feet tall.”
So, what’s the difference between the two answers they gave me regarding my height? They wouldn’t bet a modest amount of money on their first answer, but they’d be willing to bet their life on the second — even though they had no better data regarding what my height actually is. The difference was simply an ability to account for their uncertainty.
But weren’t they just guessing? After all, they weren’t using a tape measure or other common measurement device to come up with those numbers. 
Here’s a blog post that nicely captures the difference between guessing and estimating. The CliffsNotes version is simply that guessing and estimating are different points along a continuum of measurement quality. Guessing is more intuitive with little or no thought behind it, while estimation includes examining assumptions, consideration of whatever data may be available, developing rationale, and usually the use of ranges to account for uncertainty.
So by that criteria, they were probably guessing with the first value they gave me, and estimating when they used a range to describe my height. But it wasn’t just their use of a range that made it an estimate, it was the conscious selection of the minimum and maximum values, particularly in light of what was at stake (hypothetically, their life). Their mental calculation probably included a consideration that no adult human in recorded history has ever been less than an inch tall or more than ten feet tall, and it's this analytic rigor (even as crude as this example is) that largely accounts for why an estimate is more reliable than a guess.
Calibrating Estimates
Okay, so their estimate is accurate (i.e., it captures my actual height). Is it useful?
Probably not for any real decision-making (e.g., designing a cost-effective doorway that I have to fit through) because it's so imprecise. But given a few more moments to think about it, look at me again, and examine their assumptions (and maybe get input from someone else) they would almost certainly be able to narrow that range considerably. Maybe to between five-and-a-half and six-and-a-half feet tall. Without breaking out a tape measure, that might be as tight a range as they'd be comfortable with given what's at stake.
And you know what? There is absolutely nothing wrong with that.
In fact, it is not only crucial that estimates faithfully reflect uncertainty, it's also one of the most significant and often overlooked value propositions for using ranges. If decision-makers are aware of when an estimate has greater uncertainty, they can act accordingly, perhaps by choosing to be more conservative in their decisions. Or perhaps they'll choose to invest in technology, resources, or process improvements that can provide better precision over time. When all they're given is "It's Medium", they have no way of knowing whether that reflects best-case, worst-case, or most-likely case. Uncertainty isn't even part of the conversation, which leaves them blind to an important piece of decision-making information.
Regarding my second question to them about whether their answer was subjective or objective — the answer they almost always give has been “subjective”. After all, they didn’t use a tape measure to inform their estimate of how tall I am. Unfortunately, people often confuse "estimate" with "subjective". They aren't the same thing.
The most pragmatic explanation I’ve encountered is that a subjective statement expresses how someone feels about something (e.g., my favorite flavor of ice cream is chocolate), which means it can’t be proven true or false by someone else. By contrast, an objective statement is one that could be validated or invalidated by someone else.
For example, if I had instead asked the person to express whether they felt I was tall or short, their answer would have been subjective because those terms represent a personal feeling that others (maybe taller or shorter people) could feel differently about. However, because they estimated my height in physical increments which could be proven accurate (or not), both their guess and their estimate were objective in nature.
I also sometimes hear the following questions:
- "But how is an estimate a measurement?"
- "Doesn't measurement require the use of a device like a tape measure?"
As Douglas Hubbard describes in his book How to Measure Anything, measurements are simply anything that reduce uncertainty about a value or condition. Yes, it's wonderful when we're able to make measurements using a technology or device, but that's simply a more refined estimate. Technologies are never perfect either, nor are the people who use them. Many people have just adopted the position that, "If there's a technology involved, then it's not to be questioned." Regardless, the human mind is an incredibly effective measurement tool that we rely on to survive our everyday activities, and well-performed estimation is a proven method of reducing uncertainty.
Now, some readers are undoubtedly going to say, “There’s a big difference between estimating someone’s height and estimating the likelihood of occurrence or loss magnitude from a cyber event.” Well, those readers are right. There are differences, but those differences ultimately don’t matter. In my next blog post I’ll share the reasons why. Stay tuned…
