Watch the video of the Jack Jones keynote address to FAIRCON22. A FAIR Institute Contributing Membership is required – sign up now.
“We need to get our act together now,” Jack said. “One of first steps is admitting that we haven’t been doing risk measurement well so far, and if we automate what we’ve been doing, what do we get? Wrong answers faster.”
Jack focused the FAIRCON audience on the difference between trusting versus defending risk measurements. Trust isn’t always necessarily based on defensibility, as people often trust risk measurements that aren’t fundamentally sound. Risk measurement defensibility comes from:
Unfortunately, one or more of these criteria are missing from many of the risk measurements that influence cybersecurity decision-making every day.
One of the common marketing spins by those who claim to be automating cyber risk measurement is that automation eliminates human biases and assumptions. This couldn’t be further from the truth. “All risk measurement requires making assumptions,” Jack said. “An automated solution simply moves those assumptions and biases from the people doing the risk measurement to the automation solution designers, and if automation builds in wrong assumptions, then risk measurements are almost certain to be wrong.”
Jack gave several cautionary use-case examples, including assumptions based on scoring the NIST Cybersecurity Framework, a list of best practices “that was never designed as a source of data for quantitative risk analysis. This is a square hole/round peg thing.”
As part of the development of the FAIR Controls Analytics Model (FAIR-CAM™) Jack and several working groups made up of FAIR Institute members are trying to answer questions such as the ones raised in the slide below when applying NIST CSF and other frameworks to risk analysis -- and it’s a significant challenge, “the stuff of nightmares,” he said.
As important as getting to valid analysis results is presenting them. Jack’s advice:
“We have a responsibility as professionals to earn the trust of those we serve by knowing what the heck we are talking about,” Jack said.
“We have to do our homework to ensure that our measurement methods stand up to scrutiny. It is easy to come up with numbers that will look reasonable to the uninitiated, but which can’t be defended. And unfortunately, there is no cost to the people who are measuring risk poorly now. The cost is all borne by the decision-makers and stakeholders who rely on those measurements.”
FAIR-CAM will play a critical role in paving the way for automated cyber risk analysis with risk quantification. Learn more about FAIR-CAM.
Jack followed his critique of the status quo with encouraging words:
“There’s no reason for our profession to feel bad about being immature in its approach to risk measurement. Every profession evolves from lower levels of maturity to higher. There’s only cause for shame if we don’t look at this honestly and take the steps to correct it.
“In fact, it’s an opportunity. How often do people in a profession have an opportunity to make tremendous leaps in how that profession functions? It’s exceedingly rare.
“So, it’s a huge opportunity for us but it’s also a huge responsibility. We have to do our homework. We should embrace that and take it really seriously.”
Watch the video of the Jack Jones keynote address to FAIRCON22. A FAIR Institute Contributing Membership is required – sign up now.