In the first two posts of this series I talked about how most organizations seem to characterize themselves as having a “Medium-Low” risk appetite, that there is a difference between aggregate and per-scenario risk appetite, and that it is really hard to know whether risk appetite is being adhered to when using ordinal or discrete risk measurements. In this post I’ll share my thoughts on how Key Risk Indicators (KRIs) are often used by organizations in an attempt to deal with the notion of risk appetite.
KRI levels and indicator dimensions
Organizations naturally want some means of gauging when the risk landscape has changed in a way that leadership needs to be concerned about. This, of course, means understanding what “normal” looks like and drawing lines in the sand that enable recognition of at least two levels of abnormal conditions:
- “Yellow” — a warning that risk levels are worse than what is considered to be normal and acceptable, but haven’t yet reached a level of profound concern.
- “Red” — a condition representing a level of risk that is beyond leadership's appetite.
Of course, each organization will have its own definition for what Yellow or Red conditions mean, but they typically align conceptually to what I’ve described. From a risk appetite/tolerance perspective however, there are two important dimensions to these indicator levels:
- Where the thresholds themselves are defined
- How the organization responds to conditions that are in Yellow or Red states
The first of these dimensions is pretty obvious, and most if not all organizations ensure that thresholds are defined for their KRIs. We’ll come back to this a little further on. The second dimension, however, is often overlooked and treated in a more ad hoc fashion.
How aggressively and consistently an organization responds to abnormal indicator levels is actually a pretty interesting and useful expression of both its risk appetite and the amount of credibility its KRI’s and KRI thresholds have with management. When an organization doesn’t treat Red conditions aggressively, it suggests that the organization’s risk appetite is relatively high or leadership doesn’t place much credibility in the indicator (or both). For that reason, organizations should take the time to develop guidelines or policies that explicitly describe how the organization will treat these conditions. For example, an organization that wants to treat Red conditions seriously might define something like — “Any Red KRI condition will, within 24 hours, have a working group established to investigate cause(s) and develop remediation plans. In addition, senior leadership will, within 24 hours, be notified of the condition and provided a timeline for when a proposed plan of action will be available for review.” Of course, this kind of aggressive response would suggest that the Red threshold is something profoundly concerning and/or that the organization has a relatively low risk appetite.
Absent this sort of explicitly defined expectation, the level of attention many organizations apply to Red and Yellow conditions tends to vary quite a bit. It shouldn’t. Not, at least, if the KRIs are meaningful and their thresholds are defined in a manner that accurately reflects what leadership is comfortable with from a risk perspective. And there’s the rub…
An unclear understanding
Here is a short list of some of the metrics I commonly see organizations use as KRIs:
- New vulnerabilities (the results of scanning tools)
- Patching compliance levels
- Security education and awareness levels
- Malware infections
- Detected attacks
- Lost laptops/mobile devices
- Number of open “high risk” audit findings
- On-time remediation of audit findings
In this blog post I’m not going to dig into the characteristics of good KRIs, or whether some of these are good or bad KRIs. That’s an entirely different series to come (something to look forward to?). In this post I’ll limit my focus to one question — whether the risk implications for these metrics are clearly understood.
The short answer is that, with very few exceptions, organizations have not gone to the trouble of actually analyzing and measuring the risk associated with the normal or abnormal states of their KRIs. Without these analyses they have no clear, defensible notion of where to define their KRI thresholds, which means they’re probably not going to be able to justify a particularly aggressive response strategy and the impact it places on the organization. They’re just guessing. That isn’t to say their guesses aren’t appropriate; they might be exactly right. Or not. They just haven't put forth the effort to know.
The bottom line
KRIs should be an exceptionally useful tool for helping to ensure that an organization doesn’t fall completely off the rails from a risk perspective. Unfortunately, although KRIs are great at keeping organizations from falling off the rails, most organizations don’t have a clear idea of whether the rails they’re trying to stay on are where (or even close to where) they want/need them to be.
In my next post of this series I’ll back up a bit and begin laying the foundation for dealing with risk appetite more effectively. Thanks for hanging in there!
Read more:
A FAIR View of Risk Appetite - Part 1
