Most of the loss exposure in an AI system sits in parts a risk analyst can inspect: what data the system can reach, what actions it can take, and the tools it can use to take them. Unfortunately, the engineers who build these systems describe them in esoteric terms many risk analysts don't use (e.g., embeddings, context windows, tool calls, MCP servers). Does an analyst need to learn that vocabulary to run the assessment? Mostly not. What the analyst needs is the shape of the system itself: which parts exist, which of them an adversary can reach, and what the business stands to lose at each one. That is what this article is about: it's a primer about the AI attack surface in (what I hope) is easily understood terminology.
A word about sequence. Two weeks ago I wrote about assessing the risk of agentic features built into software products, and last week about the risk of using agents to write code. Both posts assumed a reader who already knew some of what this article provides. That's unfortunate. This is the foundation those two should have had. If either one felt like unfamiliar territory, that's on me.
Those posts also covered a different problem. They were about AI acting against the interests of the business with no adversary involved, which is to say an agent that does the wrong thing because it was built, prompted, or supervised badly. This post and the two I plan to follow are about someone doing it on purpose: a malicious actor.
In this post, I'll discuss the attack surface that emanates from an AI system exposure model, as depicted by the diagram below. The model includes eight stages, from the data a model learns on to the system (i.e., application) that acts on its answers. It combines two views: the model-development lifecycle across the top and the application's runtime across the bottom. Each stage is marked with what an attacker is likely to do there. The four development stages across the top are what an enterprise typically buys (i.e., procures from an AI model provider) and the four runtime stages along the bottom are the ones it configures and controls itself.
The first four stages are those where a model is built. Even though your organization likely does not train and tune its own models, it's still worth understanding these stages because they provide the opportunity for attackers to compromise a model upon which your own software solution depends.
First, the training data. A general-purpose model learns from an enormous body of text and images pulled from the public web, licensed collections, and whatever else the developer has rights to use. Nobody reads all of it, and therefore nobody truly understands what all it contains. The most concerning attack method is poisoning: content placed where it will be collected and written to change what the model learns. An attacker may need no access to the developer's network if the data is external to the developer. The loss effect is a model that behaves as the attacker intended (or not as the model developer intended) and in a narrow case that your own testing never detects. The resulting behavior may look like misalignment, but the cause is different: here an adversary intentionally created the condition.
Second, training and fine-tuning. What comes out of training is a set of weights, meaning the numbers the model "learned" from the data. Fine-tuning is the one step here that some enterprises do own: you take an existing model and train it further on data selected for your particular task or domain, changing some of its learned parameters in the process. You can't read the weights that result from training and fine-tuning the way you read source code, which is why a backdoor placed in a model isn't something an audit finds by inspection. A backdoored model behaves normally under ordinary conditions but changes behavior when some hidden trigger is included in a prompt. The trigger could be a particular phrase, token pattern, formatting convention, user identity, or even a higher-level semantic condition. The dangerous part is that conventional evaluation may show the model behaving perfectly well because the backdoor is dormant.
Third, the weights themselves and the software in which they run. A model, that is, the software code and learned/tuned weights, is an asset in its own right, and it can be stolen. The loss may include competitive advantage from the model itself and, where training data has been memorized and can be extracted, confidentiality or privacy loss associated with that data.
There may be yet another risk for model developers: rogue models and agents. For powerful models and agents, there’s potential legal liability. To illustrate, imagine if someone got hold of Claude Mythos (code and weights), disabled controls that Anthropic normally has in place, ran it on their own, and used it to hack into third-party systems. A stolen model can be used to do harm, exposing the creator to potential legal and other risks.
The software half of that stage deserves its own note. A model you pull from a public hub is a file your systems load and run, and researchers have found malicious code hidden inside models published to those hubs. The same goes for the frameworks and serving code around it. Nothing about that is new to anyone who has assessed third-party software.
Fourth, inference, which is the request your application makes when it asks the model a question. Querying a model is itself an attack technique. Enough systematic queries can reveal whether particular information was in the training data (membership inference), recover memorized information (training-data extraction), or reproduce enough of the model's behavior to build a competing model (model extraction or distillation). With this method, no intrusion is necessary. The system is simply answering the questions it was built to answer. The clearest public example of the last of these is the dispute over DeepSeek, the Chinese firm that OpenAI said "may have inappropriately distilled our models." Whether or not the allegations are true, the method OpenAI describes depends on nothing but access and patience.
The other half of the diagram includes the four application runtime stages your organization likely configures and controls. Therefore, you can do something about the corresponding exposures.
First, the prompt and the context. Everything a model sees on a given request is brought together in its context. This can include instructions written by the application's developers, the user's request, retrieved documents, and content from web pages or files the model has been asked to read. Modern AI systems can label these inputs differently and instruct the model to give some sources greater authority than others. But unlike traditional software, that separation is not an absolute security boundary (and may not exist at all in older models). The model still processes trusted instructions and untrusted content together, and malicious instructions embedded in documents or web pages can sometimes cause it to disregard the intended hierarchy. This is the fundamental problem behind indirect prompt injection.
That is the root of prompt injection, and it reflects a fundamental characteristic of how current language models operate rather than a conventional software defect waiting for a patch. Models can be told which instructions are authoritative and which content should be treated merely as data, but they cannot yet be relied upon to maintain that distinction under adversarial input. Memory can compound the problem: if malicious or manipulated information is allowed to persist beyond the interaction in which it was introduced, its influence can potentially carry forward into future sessions long after the original attack occurred.
Second, tools and the connectors that reach them. A tool call is the model emitting a structured request that other software then executes: query this database, issue this terminal command, send this message, issue this refund. The Model Context Protocol (MCP), published as an open standard in late 2024, gave developers a common way to wire those tools up. The ease of that wiring explains much of how fast the exposure grew.
Those connectors are someone else's software too. In September 2025 a package called postmark-mcp appeared on npm (the public repository developers pull reusable code from) offering an MCP server for sending email, and two days later a new version added a hidden blind copy to a domain the malicious author controlled. Every message the agent sent, attachments included, went to the attacker until the package was removed from npm later that month. ATLAS has a technique for exactly that pattern, where a clean package turns malicious on a later version.
Supply chain compromise is not a stage on the diagram. It lands at several of them: the training data, the model file and its frameworks, and the tools. So the question you must ask is: where do you import open source software or otherwise use code or artifacts that were written externally?
Third, agent identity. Here you must consider which account (identity) the agent acts as, what that account can reach, and who approved the access. Exposed items include shared service accounts, permissions inherited from a human, and standing credentials sitting in a configuration file.
Two weeks ago I argued that assessors ought to treat an agent the way they treat a privileged user. At this stage in the model, this isn't merely an analogy. The agent authenticates, holds permissions, and takes actions in systems of record. What you already know about privileged access review applies to it directly. That post also asked for three numbers on agent activity: what share of it gets reviewed, how long after the fact, and how often the review catches something. Those numbers must include tool calls.
Fourth, the consuming system. What acts on the output, and does a person see it first? An answer displayed to an employee who is free to ignore it is one risk. An answer that releases a payment, updates a customer record, or goes out to a customer without review is a different one entirely. It may be the cheapest place in the whole model to intervene.
It's tempting to name the AI system as the asset. And sometimes, this is correct. Weights can be stolen and training data can be reconstructed, so in an extraction or inversion scenario the thing at risk is the model and what went into it. A competitor who copies the behavior of a pricing model you spent two years building has taken something from you. You can put a number on the loss of competitive advantage.
More often, however, AI is part of the attack vector, but the asset is whatever leads to the business loss. Indeed, this definition of an asset is what we spend a lot of time teaching in our courses, because it can be confusing. Consider this scenario: an injected instruction causes a support agent to issue refunds to an account the attacker controls. The asset is not the AI agent that was "duped" by the injected instructions. Instead, it's the funds that were lost, exactly as it would be if an employee had been talked into the same transfer. (Note, too, that we spend time clarifying that employees are often not the threat, except when it's a disgruntled employee or a corporate spy employed by the company. Usually, when employees are involved in a scenario, they are part of the attack vector, not the threat actor.)
When the model drives a decision, the thing directly compromised may be the integrity of the decision-making process rather than data or the model itself. A poisoned or evaded fraud model loses no data at all; it approves transactions it otherwise would have declined. The resulting loss surfaces in fraud, and the assets at risk are the cash or other business resources lost as a consequence.
Let's go back to the diagram. Nearly everything in the top row is something your organization likely bought from a model provider. You probably didn't assemble the training data and train the model, and you don't have access to the weights. Fine-tuning is the real exception, and an organization that tunes a model on its own documents owns more of that row than most do. Instead, what you own in nearly every case is the bottom row: the context you feed the model, the tools you connect to it, the credentials you issue it, and the systems that act on what it says.
If you use agentic tools like Claude Code or ChatGPT Codex, you've already experienced this risk firsthand. You likely (hopefully) know that it's dangerous to give these tools unfettered access to your computer, because you never know when they'll do something bad. It's not that they "intentionally" cause harm; it's that they make mistakes, sometimes zealously pursue their goals (think: OpenAI's experience with its agents compromising Hugging Face), and can be conned into misbehaving.
This highlights an important point: buying the first half of the model (i.e., the model development) doesn't make it safe. A poisoned base model or a backdoor sitting in a set of weights would be a serious problem, and you would have almost no way to find it. Therefore, it's like many other third-party risks where you'll need to document and assess the controls the third-party has in place to mitigate them. What does the model provider do to prevent poisoning and backdoors during training? What assurance does it provide over model artifacts and its training pipeline? And, on the portion you control, how does your application distinguish trusted instructions from untrusted retrieved content, and what can the model cause to happen if that distinction fails?
Sometimes, the budget and attention you give to risk assessment go considerably further at the other end of the chain, over the stages where you have more control. In those stages, the exposure comes from things risk professionals already know how to assess: access, privilege, inputs, and what happens downstream of a decision. None of that requires a detailed understanding of how a model works.
Next week I'll take up who is being attacked this way and by what methods, using MITRE's ATLAS knowledge base and the real-world case studies it catalogs. The short version is that it is no longer only the model providers.