Quantitative Risk Management Programs (QRMP) require risk analysts to be neutral and self-aware when collecting data and estimates using the FAIR™ model. In computer science, garbage in, garbage out (GIGO) is the concept that flawed, or nonsense input data produces nonsense output or “garbage”.
How does one practice neutrality while assessing the quality of data gathered? This is one of the top conundrums when teaching the FAIR model. It begins with self-awareness in learning our own habits and cognitive biases as it relates to data gathering and calibrated estimation. Self-awareness is a skill anyone can develop with practice.
Bernadette Dunn is a FAIR Risk Trainer and a half-marathon runner.
FAIR-based risk analyses are conducted in 4 phases:
1. Scoping a Specific Scenario (identifying the asset, threat, and effect)
2. Collecting Data and Estimates
3. Running and Conducting Quality Assurance on Analysis
4. Presenting Results
One of the most valuable concepts FAIR teaches risk analysts is how to foster calibrated ranges that are both accurate and usefully precise in Phase 2 (Collecting Data and Estimates).
The 2 Types of Risk Analysts
What we typically see in the FAIR Analysis Fundamentals Training are two distinct habits among our students, when we teach estimation:
- Those that “play it safe” and produce wide ranges that are accurate but lack a useful level of precision for making informative decisions.
- Those that want to “get it right” and deliver narrow ranges aiming for precision and miss distributing a calibrated range that is accurate.
Let’s look at two examples that illustrate these distinctions:
Type 1: Play It Safe
You just moved and need to set up your internet. The provider tells you that they will arrive on Friday between 8A-5P. The contractor arrives at 4:14P. Well, the internet provider was accurate with the range they gave you for arrival; but did you sit around all day concerned about leaving in the event that the contractor shows up? What errands could you have run between 8A-4P if you knew that the contractor wasn’t going to arrive until 4:14P? This is an example of an accurate range that excludes a useful level of precision.
Type 2: Get It Right
You have dinner reservations at 6:00P. On the way out the door, your phone rings and it’s a family member frantically asking you for support. You take the time to listen and arrive at the restaurant at 6:15P. The host informs you that, because you did not arrive at 6:00P, your table was given to someone else and they are fully booked for the rest of the evening. They shared that they do give a ten-minute grace period. You do your best to negotiate with the host the nature of your circumstance and debate that you were “close” to the grace window, but the host doesn’t budge and says they literally do not have any available tables to seat you. This is an example of being given a narrow (or precise) range that denies success for achieving accuracy.
Which Habit Style Are You?
Now, it’s your turn to do a self-assessment to learn which of the two habit styles is your cognitive bias.
I'm going to ask you some questions...about me. Without taking much time to think about it, quickly write down your answers.
|
1. How tall am I? 2. How many siblings do I have? 3. What year was I born? 4. How many children do I have? 5. How many half-marathons have I run? |
The answers are at the bottom of this article to see how close you came.
But the intention of this exercise is to learn if you have the habit of precision or playing is safe with wide ranges. Here’s the checklist to assess:
- How many answers were given in ranges?
- If you provided ranges, how many were accurate (meaning the answer fell within the range)?
- If you provided ranges, how many would have been useful when describing me to someone new?
- How many answers were precise (meaning you answered without a narrow or no range)?
Give yourself (1) point for the answer falling in your range.
If you received greater than 3 points, you have a cognitive bias towards accuracy.
If you received less than 3 points, you have a cognitive bias towards precision.
Practice Calibrated Estimation
Once you assess which of the two habits you have a cognitive bias towards, you can now practice FAIR’s calibrated estimation method to build the counter-intuitive habit.
The 4 Steps for Calibrated Estimation
1. Start with an absurd range
2. Decompose the question
3. Eliminate highly unlikely values
4. Play a calibration game or equivalent bet test, revising your range iteratively until you’ve identified a 90% confidence interval.
Further details can be found in this blog post: Calibrated Estimation for FAIR™ Cyber Risk Quantitative Analysis - Explained in 3 to 4 Minutes.
Remember, to get better at anything takes practice. If you assess yourself in the group that has a preference towards precision, you need to practice step 1 and “start with an absurd range”. This will be counter-intuitive or unnatural for you which is why it will require practice.
Here are more questions to practice answering with absurd ranges:
|
A. According to a Clark School study at the University of Maryland, what is the average number of seconds per hacker attacks of computers with Internet access? B. What is the global average cost of a data breach across SMBs? C. Since COVID-19, the US FBI found what percentage increase in reported cyber crimes? D. What is the expected global cybersecurity spend in 2021? E. The number of IoT devices will reach what quantity by 2025? |
If you assess yourself in the group that plays it safe, then you want to practice steps 2 & 3. It is important to build confidence by referencing what you know and then eliminating highly unlikely values. The counter-intuitive habit to build is narrowing ranges to provide a useful level of precision.
Here are a few more questions to practice with referencing what you know and eliminating the highly unlikely values:
|
i. Unfilled cybersecurity jobs worldwide will reach what number by 2021? ii. What percentage of data breaches are caused by human error? iii. In 2018, what percentage of businesses experienced phishing and social engineering attacks? iv. In 2018, what was the average total cost for cybercrime committed globally? v. According to Comparitech, what is the average share price fall after a breach? |
Both habits require you to practice Step 4 (the calibration game or equivalent bet test). The goal in this step is to come into a range that becomes too difficult to choose between being confident in betting on your range and landing on the 90% space of a spinner wheel. If you bet on the wheel over your range, you need to widen the range. If you bet on your range over the wheel, you need to narrow your range. When you can’t decide, you are 90% calibrated to use your range. That is the FAIR calibrated estimation best practice.
Take It from Here
In conclusion, we can all improve with a little self-awareness and practice. You can use this new awareness to improve the quality of ranges you provide as well as use these tips to assess the cognitive bias when working with your SMEs. I like to encourage colleagues who are on the two opposite ends of the spectrum to work together with creating calibrated ranges that provide accuracy with a useful level of precision.
Most importantly, now that you’ve learned how to practice FAIR’s calibrated estimation techniques to improve the quality of your data inputs; you can successfully build trust with the stakeholders when answering their question: “How much risk do we have?”
Answer Key: (1.) 5’4”, (2.) 3, (3.) 1978, (4.) 0, (5.) 27, (A.) 39 sec, (B.) $3.9M, (C.) 300%, (D.) $6T, (E.) 75B, (i.) 3.5M, (ii.) 97%, (iii.)62%, (iv.) $1T, (v.) 7.27%
Learn more:
