Jack Jones wrote Understanding Cyber Risk Quantification: A Buyer’s Guide to help those looking to move up the evolutionary scale in risk management and communicate cyber risk in the non-technical, financial terms that business leaders demand.
Jack’s guide (FAIR Institute Contributing Membership required to view) includes extensive background material and specific questions to ask vendors – really all you need to know to navigate the increasingly crowded marketplace for cyber risk quantification (CRQ) solutions. Jack is the creator of FAIR™ (Factor Analysis of Information Risk), the standard model for CRQ, and is the recognized thought leader in the field.
Check out the first blog post in this series, covering Utility and Data Questions to ask solutions vendors.
Here’s a sample of the questions (with answers) you’ll find in the Buyer’s Guide section on Analytics and Reporting:
Questions Related to Analytics
Does the algorithm perform math on ordinal values?
Here’s a simple test: if numeric values such as 1-through-5 can be replaced with words (high-medium-low) or colors (red-yellow-green) then they are ordinal values, labels suitable for grouping, but not to perform math on. “In order to be quantitative, there has to be a unit of measurement — a quantity of something — like frequency, percentage, monetary values, time, etc.,” Jack has written.
Does the solution support what-if analysis?
To be truly useful for decision support, a CRQ solution should produce cost-benefit analysis for acquiring (or eliminating) controls or risk management processes. Ideally, analysis could compare the effects in risk reduction of several solutions against a baseline of risk for the status quo – and compare return on investment for dollars spent to reduce risk.
How has their model been validated?
“If the underlying analytic model is fundamentally flawed, then it doesn’t matter how good your data are — the results will not be reliable,” Jack writes. FAIR shines here: It’s been validated by The Open Group, an international body of risk analysis experts. For a quick validation, Jack also recommends checking to see that the solution doesn’t include any suspect practices (like math on ordinal values) detailed in the Red Flags section of the Buyer’s Guide.
Questions Related to Reporting
Does it break out the results into probability and magnitude components?
Loss exposure is a function of probability of occurrence and financial impact of a loss event. A CRQ solution should embrace that first principle and, Jack points out, be transparent about both factors – a necessity when “considering risk mitigation measures or other changes to the risk landscape (e.g., threat landscape changes, etc.), which might affect probability but not magnitude (or vice versa).”
Does it aggregate risk and report risk by levels (business unit, geography), risk themes (ransomware, DDoS) and other cuts at loss exposure
Quantitative analysis of cyber risk should serve stakeholders up and down the organization, from the board looking across the enterprise to the manager of single production facility – and have the capability to identify pockets of higher loss exposure to target mitigations.
How does it reflect uncertainty in results?
Another fundamental question: A good CRQ solution should generate results in dollar terms as ranges, distributions, probabilities or other reflections that risk measurement includes some uncertainty and decision-makers should see the complete picture.
Download the Cyber Risk Quantification Buyer’s Guide (FAIR Institute Contributing Membership Required)
