Jack Jones, creator of Factor Analysis of Information Risk (FAIR™) and Chairman of the FAIR Institute, delivered a keynote address to the 2022 FAIR Conference emphasizing the fact that cyber risk measurement practices have been profoundly immature, and that quantifying and automating those practices can do more harm than good.
Watch the video of the Jack Jones keynote address to FAIRCON22. A FAIR Institute Contributing Membership is required – sign up now.
“We need to get our act together now,” Jack said. “One of first steps is admitting that we haven’t been doing risk measurement well so far, and if we automate what we’ve been doing, what do we get? Wrong answers faster.”
Trusting vs. Defending Risk Measurement
Jack focused the FAIRCON audience on the difference between trusting versus defending risk measurements. Trust isn’t always necessarily based on defensibility, as people often trust risk measurements that aren’t fundamentally sound. Risk measurement defensibility comes from:
- A clear scope of what’s being measured,
- Accurate and relevant data,
- Models that are logically and formulaically sound, and
- Results that faithfully reflect the measurement’s uncertainty (using ranges & distributions).
Unfortunately, one or more of these criteria are missing from many of the risk measurements that influence cybersecurity decision-making every day.
Cyber Risk Measurement Automation – Watch Your Assumptions
One of the common marketing spins by those who claim to be automating cyber risk measurement is that automation eliminates human biases and assumptions. This couldn’t be further from the truth. “All risk measurement requires making assumptions,” Jack said. “An automated solution simply moves those assumptions and biases from the people doing the risk measurement to the automation solution designers, and if automation builds in wrong assumptions, then risk measurements are almost certain to be wrong.”
Jack gave several cautionary use-case examples, including assumptions based on scoring the NIST Cybersecurity Framework, a list of best practices “that was never designed as a source of data for quantitative risk analysis. This is a square hole/round peg thing.”
As part of the development of the FAIR Controls Analytics Model (FAIR-CAM™) Jack and several working groups made up of FAIR Institute members are trying to answer questions such as the ones raised in the slide below when applying NIST CSF and other frameworks to risk analysis -- and it’s a significant challenge, “the stuff of nightmares,” he said.
Advice on Presenting Quantitative Risk Analysis Findings
As important as getting to valid analysis results is presenting them. Jack’s advice:
- Understand the assumptions you’re making and be prepared to explain them.
- Welcome skepticism and challenges to your measurements.
- Feedback and discussion are how we improve.
- Remember that the goal is accurately informed decisions, regardless of how that occurs.
- The process of getting to results is often just as valuable as the results themselves.
“We have a responsibility as professionals to earn the trust of those we serve by knowing what the heck we are talking about,” Jack said.
“We have to do our homework to ensure that our measurement methods stand up to scrutiny. It is easy to come up with numbers that will look reasonable to the uninitiated, but which can’t be defended. And unfortunately, there is no cost to the people who are measuring risk poorly now. The cost is all borne by the decision-makers and stakeholders who rely on those measurements.”
FAIR-CAM will play a critical role in paving the way for automated cyber risk analysis with risk quantification. Learn more about FAIR-CAM.
The Challenge Ahead for Quantitative Cyber Risk Management (CRQ)
Jack followed his critique of the status quo with encouraging words:
“There’s no reason for our profession to feel bad about being immature in its approach to risk measurement. Every profession evolves from lower levels of maturity to higher. There’s only cause for shame if we don’t look at this honestly and take the steps to correct it.
“In fact, it’s an opportunity. How often do people in a profession have an opportunity to make tremendous leaps in how that profession functions? It’s exceedingly rare.
“So, it’s a huge opportunity for us but it’s also a huge responsibility. We have to do our homework. We should embrace that and take it really seriously.”
Watch the video of the Jack Jones keynote address to FAIRCON22. A FAIR Institute Contributing Membership is required – sign up now.
