Let’s start with the questions regarding the utility of CRQ and the data that’s necessary.
How does the CRQ solution define the “risks” being measured?
The most important question to ask, Jack writes, fundamental to choosing an effective quantitative risk analysis solution: Risk must be analyzed as loss-event scenarios with measurable probability of occurrence and size of impact.
Here’s an example of a scenario statement, with the necessary elements of a threat actor impacting an asset in some way:
“Analyze the risk associated with malicious external actors breaching the confidentiality of sensitive company data accessible on a lost/stolen mobile device.”
Simple and straightforward. And yet, as Jack writes in the Buyer’s Guide, “It’s common to see solution providers confuse and conflate” things from the risk landscape with risks, such as identifying “weak passwords” or “disgruntled insiders” as “risks” – they are not, because they’re not loss event scenarios.
What parts of the risk landscape does their solution analyze?
CISOs often have responsibilities that branch out from cyber risk to technology and operational risk, so it’s important to understand exactly the focus (and limits) of a quantitative risk management solution. Here’s a sample:
>>For assets, the solution may cover data but not IT facilities
>>For threats, the solution may cover malicious outsiders but not technology failure
>>For outcomes, the solution may cover data breaches but not regulatory compliance failures.
What assumptions are made in applying historical data?
The cyber risk landscape is highly dynamic – how do vendors account for the fact that past experience may not reflect the future? How do they account for the differences in loss-event impact data among industries – can you customize the solution to your industry or business model?
Where does control-related data come from?
The wide variety of sources for controls data – GRC’s, frameworks, vulnerability scans – is “a blessing and a curse.” As one example, Jack writes, controls descriptions in the popular frameworks “have not been defined carefully enough to be clear on how to apply a control measurement within risk analysis.” Also, none of the frameworks adequately describe the roles of controls or the dependencies among them (a problem Jack addresses in his FAIR Controls Analytics Model or FAIR-CAM™).
How is measurement uncertainty accounted for?
“Because we don’t have a perfect understanding of the variables that affect the future, there will always be some amount of uncertainty,” Jack writes. Solutions that generate precise or single-number outputs don’t reflect reality. Instead, look for results displayed in ranges, best case/worst case/most likely or other distributions.
Get all the details on smart shopping for a cyber risk solution. Download Understanding Cyber Risk Quantification: A Buyer’s Guide.