In my previous post (No Data? No Problem) I discussed the question, “How do you make estimates when you have no data?” This post focuses on a related question – whether historical data can be relied upon to reflect the future.
A flawed assumption regarding this concern about historical data is that you would only use historical data in making estimates. This shouldn’t even remotely be the case. For example, if I’m evaluating the risk associated with DoS (denial of service) attacks against my company’s website, I will certainly leverage historical data to inform my risk estimation. I will also take into account any trends that historical information may suggest are occurring, as well as other considerations that might affect how the future unfolds, such as:
- Has my company done (or is it considering doing) anything that’s likely to increase its value as a target?
- Are new DoS technologies/methods evolving that might increase the frequency or severity of attacks?
- Has my company improved its capabilities for resisting or minimizing the impact of such events?
- Are there other aspects of the risk landscape that might make the future of DoS attacks materially different than the past?
Whenever possible, I will leverage expertise within my network of contacts and colleagues (e.g., threat intelligence experts) to examine these and similar questions to improve my estimates regarding the future.
Keep in mind that when it comes to risk analysis there is never a question of whether data is perfect – your data is never perfect. Risk analysis is, however, always a matter of effectively leveraging the data you have to reduce uncertainty about the future as much as you can given your available time and resources. Furthermore, it's important to recognize that there are diminishing returns, in terms of analysis quality, from more data — i.e., the 100th piece of data does not reduce uncertainty as much as the first few. Douglas Hubbard nails this as well, within his books.
The bottom line is that historical data can and should be leveraged, but always with due consideration for how the landscape appears to be evolving. BTW – my white paper on "Effectively Leveraging Data in FAIR Analyses" also discusses the question of historical data, and is available on the FAIR Institute members resources page.
In the last post of this series, I’ll walk through a “data challenged” risk analysis that includes the question of historical data. Stay tuned!
Related:
Measuring DDoS Risk Using FAIR [Video]
