I’ve heard critics of quantitative risk analysis challenge the approach, stating that it is “too difficult”, “time consuming” or that their organization is “simply not mature enough for quantification.” In my experience, a majority of such arguments can be addressed by revisiting a few fundamental FAIR concepts. In this blog post, I will address the notion of analysis paralysis and how the amount of precision and rigor in the analysis is driven largely by the objective of the risk analysis (e.g., the questions the analysis will inform). If decision makers understand this, they can also help to encourage their analyst to weigh the cost/benefit of investing additional time/resources in their analysis if the results are ultimately “good enough” to meet the stated objective.
In FAIR, we call this achieving accuracy with a useful amount of precision.
The Open Group defines accuracy for risk analysis as “our capability to provide correct information”, while precision is “exact, as in performance, execution, or amount”. Using distributions or ranges can bring higher degrees of accuracy to estimates and also helps to account for uncertainty by combatting estimates that are precise but may miss the mark (“precisely wrong”). The “useful” piece is ultimately left to the discretion of the decision maker and could also vary based on the decision that you are trying to inform.
The image below illustrates this concept:
To further support this concept, below are two scenarios organizations may face where the degree of accuracy differs based on the stated objective.
Scenario 1: Prioritization of Top 10 Cyber Risks - Accuracy with Less Precision
Prioritization of an organization’s top security risks first requires that each risk scenario identifies how often each bad thing is likely to happen (probable loss frequency) and how much money is at stake when that bad thing actually happens (probable loss magnitude).
Certain perceived top risks can quickly be eliminated by performing a very high-level FAIR analysis using wider, calibrated ranges. For example, I’ve seen organizations allow the loss magnitude side of the FAIR model dictate their top 10 list; however, if the frequency of the event is very low that may be a way to rule these out.
Another tip to keep in mind is that when you are reporting top risks in terms of dollars and cents using wider (less precise) ranges: This is still a notable improvement from simply labeling all 10 risks as high or “very high”.
Using this mindset can also help analysts who are experiencing “analysis paralysis”, a scenario where the time and resources invested into the analysis do not provide material benefit in the analysis output.
The FAIR Institute was named one of the three Most Important Industry Organizations of the Last 30 Years by SC Magazine.
Scenario 2: ROI Analysis over a Significant Business Decision - Accuracy with More Precision
In this scenario, the objective of the analysis is to help determine whether a significant business decision is worth the cost of investment. For example, let’s take a scenario where an organization is looking to justify the increased expense – about $100,000 a year—in moving customer data to the cloud-based Office 365 from their internally hosted Exchange Server.
In order to help determine whether the migration would help achieve better security and lower risk, a series of risk analyses had to be performed. Specifically, a current state analysis of the amount of risk associated with sensitive data becoming exposed in the internally hosted Exchange server compared to a second analysis showing the probable future level of risk if the organization moved to a cloud-based email solution.
In this scenario, additional rigor and precision is likely required to be able to justify the cost of the investment. For example, tighter ranges were likely used when obtaining estimates related to controls in place to prevent breaches (i.e., current patching process, email filtering capabilities and encryption abilities) so these could be compared against the probable improved security available from O365. With a quantified understanding of the impact of a breach of sensitive customer records, executive management confirmed it would be a smart business decision to move to the Office 365 suite hosted in the cloud. For additional information, view the entire Office 365 case study.
In conclusion, if you are ever experiencing analysis paralysis, take a step back, revisit the original objective of the risk analysis and ask yourself if the degree of precision is appropriate to inform the stated objective. And remember, don’t let perfect get in the way of good enough!
More advice about getting over analysis paralysis:
Warren Buffet's Information Security Advice
How to Filter Out 'Fake Risks' from Your Risk Register
Coming September 24 & 25: the 2019 FAIR Conference, the year's big event for learning the most advanced techniques in risk management -- and networking with the best thinkers in the risk profession.
