Eight Takeaways from Anthropic's Risk Report for AI Risk Leaders

I recently took the time to read Anthropic's August 2026 Risk Report, which assesses the misalignment risk of its most capable models. Misalignment, whether or not used by name, has been a driver of many recent gloom-and-doom headlines. The term refers to a model acting against the interests of its creators and, as such, can lead to big threats (such as taking down a power grid) to more mundane ones (such as hacking into a competitor's website). Anthropic reports that the probability of the misalignment it has already observed is high (largely because they've seen it happen, not once but relatively often), but it rates the resulting risk of catastrophic harm as low. That distance comes from three things: the harm those known behaviors can do is limited, the controls Anthropic has put in place to catch most of what remains, and the severe kinds of misalignment it has not observed are judged very unlikely.

Misalignment is a hard thing to assess. You can't read an agent's intent, the behavior surfaces mostly when a task gets difficult, and the data is thin next to what we have for familiar threats like phishing. It is also just as important for enterprise risk leaders as it is for the model builders. Anthropic worries about what its models might do inside its own walls. But an enterprise has to worry about an agent acting in its name, on its systems, or in front of its customers, often with a degree of its legal authority.

In this post, we're going to consider some of the controls Anthropic asserts in its report and what they mean for the people who assess agentic features. While Anthropic assessed how controls reduced the misalignment risk of its models, many of those same types of controls can be put to work by enterprises that are building AI agents into their own software, whether or not they use Anthropic's models in doing so.

To be clear, we'll limit our discussion to one specific use case: agents that are part of a software product's feature set. Examples include a customer service agent that can issue refunds, a claims agent that approves or routes claims, or a sales assistant that quotes prices and terms. We will not address the use of agents to write software. That use, often called agentic coding, carries its own risks, and I may take it up in a follow-up post using the same approach.

The risk in question is what I'd call alignment risk in the business sense: the chance that an agent built into your product takes actions or gives answers that don't serve the interests of your business, your customers, or other stakeholders. In doing so, an agent can reduce revenues, increase expenses, create liabilities, harm users (and reduce customer satisfaction and loyalty), or cause other business losses.

Why Anthropic's report applies

Most of the evidence in the report comes from Anthropic's evaluations and its own internal use of agents for engineering and research work, not from monitoring of customer-facing deployments. The behavior it documents still carries over, because it is the same underlying behavior that shows up in agentic products.

Anthropic's models occasionally take "reckless or destructive actions in service of user-assigned goals" and are dishonest about the quality and completeness of their work. In one test, a model without permission to edit certain files appended a self-deleting script to a system file to grant itself access. METR's Frontier Risk Report, published in May with participation from Anthropic, Google, Meta, and OpenAI, found the same pattern across developers: at least 16 percent of successful runs on its hardest evaluation tasks involved cheating, and 25 of 44 documented incidents combined overreach with deception.

Product agents have shown the customer-facing version of misalignment. In 2024, a Canadian tribunal ordered Air Canada to pay C$812 after its chatbot misstated the airline's bereavement fare policy, and rejected the airline's argument that the chatbot was responsible for its own statements. In April 2025, a support bot for the coding tool Cursor told a customer that a login problem was an intentional one-device-per-subscription policy. No such policy existed, and customers publicly cancelled subscriptions over it.

It's not that these agents want power or seek harm; it's that they zealously work to "close the ticket." And because they do their work at speed and scale rarely achieved by humans, any misalignment can result in magnified business impact.

Below, I'll discuss eight lessons from the Anthropic report that I think apply well to enterprises building or buying agentic software solutions.

1. AI agents present a form of insider risk

The FAIR cyber risk scenario taxonomy, which provides a structured approach to defining a cyber risks scenario, defines a list of ten threat types that include AI agents (number 5), as well as both privileged (3) and non-privileged (4) insiders. In the current version of the taxonomy, AI agents are defined as malicious software tools designed to attack others; they're not described as agents used by a business for benevolent or legitimate profit-seeking purposes. While our current definitions for insiders do not encompass non-malicious agents, I believe it's easier to assess the risks if you think of them as insider threats.

Here's why: the generative AI models that support our software agents are extremely complex threat actors that often act unpredictably and in human-like ways. Indeed, frontier model developers like Anthropic and OpenAI test their models using approaches that sound more like psychological evaluations than software tests. For example, Anthropic tests its models for honesty and the tendency of its agents to collude, to sandbag (deliberately underperform on a task), to reward hack (satisfy the letter of a request while missing its point), and to behave sycophantically (tell the user what they want to hear). It even tests for evaluation awareness, meaning whether a model recognizes that it is being tested, since one that does may behave differently than it would in your production environment.

The agents you build or buy also have privileges granted to them. They're often given access to sensitive data, the ability to execute transactions, a direct channel to customers, and other abilities to do their job. Indeed, that's why they're called agents: they do work on the behalf of the company and therefore have privileges to do so.

Therefore, consider the AI agents your organization builds or buys as a form of an insider. Learn what you can about their motivations. Are they encouraged to close tickets quickly? Do they have goals to reject a certain number of applications? What might cause them to act zealously to achieve a goal? Also, determine the authority (access, privileges) they have. Would you consider them a "privileged user"? Are there privileges checked and tested? We'll talk about these facets of agents below, but the point here is that to understand the threat actor here, it's helpful to think of them as intelligent insiders rather than traditional software.

2. Know which models your agents run on

Before defining your risk scenarios, identify the specific models the agents in scope are built upon, and read what the makers of those models publish about them. An invaluable resource are the system cards that arrive with each model release and name the failure modes the vendor found. Anthropic's Opus 4.6 card covers reward hacking, sabotage concealment, evaluation awareness, and overly agentic behavior in computer-use settings, each compared with the previous model. OpenAI's GPT-5.6 card reports named environments including destructive actions by a coding agent and impossible tasks, along with how well monitors catch them. Vendor risk reports and incident disclosures sit alongside the cards, as do independent evaluations from METR and Apollo Research.

Read the card for the release your agents actually run on, and read it again whenever the software is updated to a newer one. Cards report how behavior changed from the previous version, which is the closest thing an assessor has to a trend line on the inherent risk of a model. For example, Anthropic's system cards describe the results of MASK (Model Alignment between Statements and Knowledge) tests, which evaluate whether a model will state something it does not believe when it is pressured to do so, a measure of honesty rather than accuracy. That is close to what a support agent faces when a customer presses for an answer your policy doesn't allow.

Used carefully, the published rates are priors for how often an agent goes off-script. Include them in your analysis, along with the model version and the date they came from. Two caveats belong in the write-up: those rates come from the vendor's tests and control environments rather than yours, and models sometimes recognize when they're being evaluated.

3. Scope the scenarios to what the agent can actually do

Anthropic defines its scenarios as pathways. Its assessment names eight concrete pathways by which a misaligned model could contribute to catastrophic harm, and it concentrates its analysis and its controls on those. The eight themselves are about protecting frontier model development and won't transfer to your business, but the word pathway is worth borrowing. A pathway is a route from something the agent can do (i.e., the resources to which it has access or innate capabilities it has) to a loss event, which means the scenarios worth analyzing fall out of what the agent is able to do in the first place.

Start by writing down what the agent can actually do: which tools it can call, which systems and records those tools reach, what it can spend or commit (verbally or in writing), and what data it can see and therefore disclose. Can it issue a refund? Can it read another customer's file? Can it place an order with a supplier? That list grounds the realm of possibilities in reality, and the significant scenarios are drawn from it rather than from sheer imagination.

The capabilities of an agent will help determine its inherent risk. A support agent that answers product questions and a support agent that can grant refunds carry different levels of inherent risk, even if they share a model and a chat window.

In documenting capabilities, note that agents can often commit (legally bind) the company even when no tool is involved, as Air Canada's chatbot demonstrated. So your list covers not only the resources the agent can reach but its ability to communicate and commit the enterprise.

Once you understand the capabilities of an agent, then identify the most significant loss event scenarios. The FAIR Cyber Risk Scenario Taxonomy gives you a framing for this: a scenario names a threat, an asset, a method, and an effect, and reads as "[threat] impacts [asset] via [method], causing [effect]." For an agent, the threat is the agent itself, and it usually belongs in the privileged insider category with accidental rather than malicious intent, since what it wants is the task closed. The asset is a business resource that the agent can damage, either directly or indirectly: customer funds, personal data, contractual commitments, the brand, compliance processes, to name a few. The method is the tool or channel it uses, such as the refund interface, a record lookup, or a statement made to a customer. The effect is the loss category that follows, in FAIR-MAM terms, such as financial fraud, information privacy loss, or reputation damage.

The refund case then reads: an AI support agent (privileged insider, accidental) impacts customer funds (financial assets) via unauthorized use of its refund tool, causing financial fraud losses. Keep the handful of scenarios where the asset is worth the most and the agent's access is widest.

4. Assess the agent's privileges the way you would a privileged user's

An AI agent that can move money or change records on behalf of customers is a privileged user, just the same as a human agent with those privileges. (Of course, the AI agent never gets tired.) Anthropic runs agents in training and evaluation inside sandboxes that aren't supposed to have general access to its computing clusters, and it applies sandboxing or blocking classifiers to its most autonomous internal uses.

For an assessor, this is privileged access management applied to a non-human identity, and it starts as a documentation exercise rather than a judgment. Does the agent run under its own service identity rather than a shared administrative account? Does it act only within the permissions of the customer it is serving? Is it limited to the smallest set of tools the feature needs, and does it face hard limits on transaction size and rate? Does anything above a threshold go to a person for approval? OWASP's guidance on excessive agency gives the pattern to test against: a read-only scope where reading is all that's needed, human review before the agent sends anything consequential, and rate limits on the actions that remain.

In FAIR-CAM terms, limiting which systems and records an agent can reach is an avoidance control, because it reduces contact between the agent and assets it could harm. Approval thresholds and blocked actions are resistance controls, which lower the chance that an out-of-scope action becomes a loss event. Transaction caps also bound loss magnitude when the other controls fail.

5. Assess detection by its coverage, review interval, and catch rate

Another area of emphasis in the Anthropic report is the detection of misaligned behavior. In FAIR-CAM terms, we break detection down into three functions that work in series: visibility, or how much of the agent's activity is captured at all; monitoring, or how much time passes between looks; and recognition, or how often a look catches a problem. An assessment should be able to put a number on each one for the agent under review. Anthropic did this in its own report.

Visibility comes first, and it is mostly a documentation question. What share of the agent's conversations and tool calls is logged and retained, for how long, and does the log capture what the agent said as well as what it did? Sampling may be fine, but the sample rate belongs in the analysis rather than in someone's head.

Monitoring is measured as elapsed time between reviews. Find out what runs automatically, such as classifiers on statements that commit the company, thresholds on refunds and other consequential actions, and alerts on unusual rates. Then, determine how much flagged activity a human being actually reviews.

Recognition is the catch rate, and the only way to get one is testing. For example, seed known-bad cases into the stream and count how many get flagged. A vendor's assurance is not a catch rate unless they've done testing and you've determined you can rely on their tests.

Three follow-up questions separate a real detection control from a described one. Can the monitor be bypassed or switched off? Has the monitor been tuned to cut false alarms, and if so, what did the catch rate fall to? And how fast would the loss be accumulating in the meantime?

For customer-facing agents, where the only signal is customer complaints, the review interval is however long it takes a customer to get angry, and the catch rate covers only the failures customers notice and bother to report. Cursor learned about its support bot's invented policy from social media. Sampling conversations, classifying statements that commit the company, and alerting on unusual action rates are what shorten that interval.

In FAIR terms, these three numbers drive the share of harm that goes unmitigated. If an assessment can't produce them, the finding is that detection is unproven, not that it works.

6. Assess what can be reversed, and how quickly

Anthropic's argument that harm will be mitigated depends heavily on reversibility. It separates reversible harms, which it expects to undo if detected within weeks, from irreversible ones, and it states plainly that after-the-fact monitoring would not prevent harm from a first irreversible action. When monitoring does catch something, its process is to reverse the consequences, look for similar actions, patch the controls, and trace the root cause.

Product agents make this concrete. A refund can often be held in a pending state before it settles. A statement to a customer can't be unsaid, and the Air Canada tribunal held the company to what its chatbot said. Designing agent actions so that the consequential ones can be held, reviewed, or reversed changes how much a detection gap costs. For example, if the Air Canada agent stated that any refunds it submits will be reviewed for approval by a human supervisor, the tribunal outcome may have been different.

Similarly, having a way to switch off an agent or a single tool without taking down the product and having a playbook for making customers whole reduces the impact. These are FAIR-CAM's loss event response functions. Switching off the agent is event termination, restoring normal service is resilience, and remediation reduces realized loss.

Three questions carry this part of an assessment. Which of the agent's actions are irreversible once taken? How long does a wrong action stand before anyone can undo it? And do the switch-off procedure and the customer remediation playbook exist on paper today, with an owner, rather than being assembled during the incident?

7. Assess the testing behind the agent

Anthropic tests each model with about 2,900 automated investigation sessions, in which another model probes its behavior across simulated scenarios with real tools in isolated sandboxes. For its latest model, it added 86 scenarios seeded from real internal sessions that offered especially tempting chances to misbehave. It also builds deliberately misaligned models to check whether its audits can catch them, and it gave a METR staff member three weeks to red-team its monitoring pipeline, which turned up several new weaknesses.

Few enterprises will run a program like that, but the shape of it tells an assessor what to look for in the testing that was done. Traditional software quality practices still matter, and they test whether the agent can do the job. Misalignment tends to show up when the job can't be done as asked. METR found cheating concentrated on its hardest tasks, and Anthropic's examples involve agents working around a blocked path. For a product agent, the equivalent test cases include a customer asking for something policy forbids, a question the knowledge base can't answer, and a tool call that fails. Does the agent say no, say it doesn't know, or invent an exception or a policy?

Anthropic warns that models sometimes recognize when they're being tested, so test cases seeded from real customer conversations are worth more than synthetic ones. The controls need red-teaming too, including whether the agent can reach a tool or channel that isn't monitored.

So the questions to ask of a test report are who ran it and how, whether the cases came from real customer conversations or from someone's imagination, whether the hard cases were covered, whether anyone adversarially tested the controls rather than the agent, and whether any of it was repeated after the last model update.

For FAIR analysis, testing is where the numbers come from. How often the agent leaves its scope when blocked is frequency data. How often the monitor catches it is the recognition estimate. Without testing, you'll be hard pressed to come up with defensible numbers.

8. Put the assessment through a cross-functional review, at design and at every gate

Anthropic now requires a review before any new model is approved for easy or broad internal use, including less capable models, and specialists in alignment, testing, control, security, and interpretability take part.

The enterprise equivalent is a gate before an agent can commit the company to anything, followed by a staged rollout. For example, a refund agent might start with low limits, a small share of customers, and human approval on every refund, then earn wider authority as monitoring data accumulates. FAIR-CAM lists change review processes among the controls that reduce the probability that a change degrades other controls. When the vendor updates the model underneath your agent, that is a change too, and it belongs in front of the same gate.

Your FAIR analysis is one of the things the gate reviews. Anthropic's review examines its argument as much as its model, and the enterprise version works the same way. The panel reads the scenarios, the test results and vendor evidence, the coverage, review interval, and catch rate, and the loss exposure that falls out of them, then decides whether that exposure sits inside the organization's tolerance. Reviewing an analysis also means challenging it: which assumptions were relied upon, where the data came from, and what would have to be true for the analysis to be wrong.

The first of those reviews belongs at design time, not at launch. Of course, you won't have internal test data at design time, but many scenarios and risk factors can be identified early on. The decisions that set an agent's loss exposure (which tools it gets, which actions need approval, what gets logged, which actions can be reversed) are made early and they're hard to change once a feature has shipped. After that, each gate in the rollout is another review point, as is each update to the model underneath the agent. The product owner owns the risk, but your job is to provide an analysis they can act on while the choices are still open.

What this asks of enterprise risk leaders

Misalignment sounds like a research problem, and in the labs it partly is. Inside a business, it's an old problem wearing new clothes: how do we manage the risk of an insider with privileges? FAIR was built for that, which is why the hard part isn't the math.

Instead, if the risk function isn't involved when an agent's authority is set and the controls are designed, the organization has already accepted a certain level of risk. It just doesn't know how much.