While I could easily write a novella focused solely on the many benefits of FAIR over other risk analysis methods, there is an important caveat to keep in mind when conducting FAIR analyses: The analysis is only as reliable as the analyst who conducted it.
Think about it: FAIR is more than a quantitative risk model, it is a process for analyzing risk. If the process is not followed, then the results can be worse than useless. They can be misleading.
So, it’s on FAIR risk analysts to constantly check their processes against best practices, check their data with subject matter experts, check their assumptions and conclusions with stakeholders – and check their biases at the door.
In that spirit, here are my 4 Tips for Reality Checking a FAIR Analysis:
1. Properly Scope the Analysis
When conducting a FAIR analysis, the first step is to scope the scenario. This involves clearly articulating the asset, threat, and effect.
A properly scoped scenario is the difference between vague ("rogue insiders") and specific ("the risk associated with a malicious privileged insider breaching the customer information contained in database X").
Imagine trying to provide data related to "rogue insiders". Some subject matter experts may be assuming you mean causing an outage of the key application, others might assume you mean stealing data, and some might just be downright offended (DBAs don't often like this implication).
An improperly scoped scenario leads to confusion, unstated assumptions, and inaccurate data collection.
2. Ask the Right Questions, the Right Way
The second step of conducting a FAIR analysis is data gathering. This is the point where the analyst relies on data from subject matter experts in order to create estimates for each of the components of the model.
To gather data effectively, you must do the following:
Create and ask context specific questions that relate the data point required directly to the model. For example, “how often do we see attempts by cyber criminals to breach the customer data contained in database x?”
A properly worded context specific question includes the asset, threat, and effect. If you were to ask instead "how often do we see attempts to breach database x?" you're likely to get different (and higher) estimates because you did not specifically instruct the subject matter expert to include only those attempts by attackers to get the customer data.
Use calibration to help subject matter experts provide usefully precise ranges with 90% confidence. A fundamental concept in calibration is "usefully precise" estimates.
If you ask a subject matter expert the above question and they respond with "between once in fifty years and ten times per year" then the odds are they are probably accurate, but is that usefully precise? On the other hand, if they respond with "3.7", is that accurate? Using calibration, you can help the subject matter expert provide a range that is both usefully precise and accurate.
Thoroughly document the rationale related to all data points, such as the name and title of the person who gave you the estimate, the data they used to inform it (historical events, logs, external sources, etc.), controls considered, any sources of remaining uncertainty.
Unless you have a photographic memory, rationale is the only way you are going to remember the important details of your estimates. Failing to do so may make follow-up questions difficult or impossible and it harms the credibility of your analysis if you are unable to defend your estimates.
3. Perform an Initial Results Review with Key Data Sources
One of the reasons FAIR analyses are defensible and rigorous is that they are not done by one analyst in a black box, but rather are a combined effort involving multiple subject matter experts from various areas of the organization.
The efforts of the various subject matter experts can easily be wasted if the analyst fails to do a preliminary result review and Q/A with them. This allows for a “gut check” of the results as well as an opportunity to reevaluate estimates and assumptions and verify if additional data is needed.
For example, it is possible to mis-key a value when entering it in an analysis--doing so could be the difference between $10,000 in loss exposure and $1,000,000 in loss exposure. However, this detail might not be caught if the results are not properly reviewed.
Or, another frequent occurrence is inaccurate assumptions related to threat event frequency and vulnerability. When an event that has never occurred in the history of the company comes out with a probability of occurring five times in the next year, there's a good chance there is an incorrect assumption somewhere. This is the time to revisit the estimates with the subject matter experts and verify if they appear to be reasonable.
4. Be Prepared to Defend the Analysis
The final step of the analysis is to present the results. This is where all of the hard work culminates into a 30 – 60-minute presentation.
When presenting quantitative results, especially to stakeholders used to seeing qualitative results, you are likely to get a lot of questions around how the values were determined and what goes into the amount of loss exposure shown. This is good! It means you have the interest of the audience. It also means you must be prepared.
In order to best answer questions and defend your analysis, be prepared to provide the rationale related to any of the estimates in the analysis. For example, the most frequent question I hear is "how did you come up with once in five years to once a year as your range for threat event frequency" or "what information do we have that says we are 50-75% vulnerable to this type of event?" It is good practice to have appendix slides prepared with the rationale related to key data points in the analysis, in the event they are questioned.
Also, keep your audience in mind. What is their “so what?”. If your audience is more focused on the current-state risk, it might not make sense to focus on a bunch of different control improvements and their ROIs. Likewise, if the audience is interested in how to remediate a control finding, then those ROIs are likely important to the presentation. Make sure you gear your presentation toward the goals and interest of the audience to guarantee you provide value in your results.
As with any model, FAIR is only as strong as the that analyst using it. Make sure that everyone in your organization using the FAIR model is appropriately trained to do so. By leveraging trained analysts and utilizing the process and tips outlined here, you will be on the path to creating rigorous, defensible, quantitative FAIR analyses. Learn more about FAIR training.
