The Two Perils of AI Risk: What the Model Says vs. What the Model Sees

Enterprises talk about "AI risk" as one thing. Insurers are discovering it is at least two, and the two behave so differently that managing them together makes both harder.

What follows came out of a public argument. Over several weeks, practitioners pushed back on things I had written: Barry Rabkin, who has spent 45 years analyzing the insurance industry; Pratap Tambe, a cyber risk expert; Avinash Chauhan, an insurance product manager; James Clark Mitchell, an enterprise risk consultant; David Wray, who works with global financial services institutions; CJ East, a data engineer; and Vinay Bansal, who builds verifiable infrastructure. Several of them disagreed with me. Some still do. The framework below is better for it, and I have kept their objections in rather than smoothing them out.

The claim that was too loose

I wrote that architecture-constrained AI can be underwritten. Barry's response was direct: no underwriter should accept that claim unless the technology firm making it will pay every dollar of every lawsuit arising from a hallucination that reaches a commerce-facing process. Expecting architecture to eliminate hallucinations, he said, puts users in the position of the naked emperor.

He was right. Nothing architectural makes a non-deterministic model deterministic. Any vendor claiming their guardrails eliminate output errors is selling exactly what he described.

But the exchange exposed something the AI risk conversation keeps missing. The claim was wrong about one peril and right about a different one.

Peril one: what the model says

Output risk is the risk that a model produces a wrong answer that causes harm. A hallucinated figure in a client deliverable. A fabricated citation in a filing. A confident recommendation someone acts on.

This peril is hard to bound. Large language models are non-deterministic by construction. The same prompt can produce different outputs, and the failure mode is not a crash but a plausible error that survives casual review. Jenn Tejada, PagerDuty's executive chair, described the agent version of this: when AI drifts, it is harder to see, and you often do not see it until it has already executed that drift in several ways, at which point it has become multiple failures.

Apply the standard underwriting test and this peril fails it. An insurer must be comfortable estimating the maximum probable loss of what it sells. Nobody can credibly estimate the maximum probable loss of what a model might say across an enterprise's workflows. Barry's conclusion follows: if you cannot estimate it, the fiduciary answer is not to sell the coverage. A risk existing has never obligated anyone to cover it.

He adds a point that deserves more attention than it gets. The plaintiff's bar is the wildcard that maximum-probable-loss estimates tend to omit. Defense costs and social inflation mean an insurer's true cost routinely exceeds the stated limit, and applied to output that is wrong in ways nobody can enumerate in advance, the tail is hard to see the end of. That is the strongest form of his argument, and I have no rebuttal to it.

The mitigations for output risk are real but they are not architectural. They are scope, keeping non-deterministic output away from decisions that cannot tolerate being wrong; human decision authority that stays meaningful rather than ceremonial; and verification that catches errors before they execute. Stacking more probabilistic filters onto a probabilistic system does not produce determinism. It produces a better demo and a slightly different error distribution.

Peril two: what the model sees

Exposure risk is the risk that sensitive data reaches somewhere it should not: the model provider's systems, its logs, its training data, another tenant, an attacker.

This peril behaves nothing like the first. It is not probabilistic. It is set by architecture, and architecture is knowable in advance.

If a system is built so the model receives the structure of a question and never the underlying values, and the data is encrypted before it leaves the customer with keys the customer holds, then exposure through that model is bounded the way a vault bounds physical loss. You can state exactly what was reachable, and the statement holds without trusting anyone's intentions. This is the architecture we described in What Is Confidential AI and walk through step by step in How to Run AI on Sensitive Data Without It Leaving Your Control.

David Wray put the enterprise version of this better than I had: most large organizations are not looking for perfection. They want clear boundaries, controls, and evidence that the controls work. That is a far easier conversation to have with a risk team than "trust us."

A promise not to look and an inability to look sit in different categories for an underwriter. One can be priced. The other has to be taken on faith.

The two perils side by side

Both perils are real. They are managed by different people, with different tools, and only one of them has a cap available today.

How an exposure bound actually works

What about zero data retention?

CJ East asked the technical version of the objection that matters most: if a provider offers zero data retention, is that not enough?

It is a real control, and the providers offering it are serious about it. It closes the retention and training vectors, which is most of what people worry about.

What it does not change is that the data was processed in plaintext on someone else's infrastructure. Retention is a commitment about what happens afterward, and it is a commitment you cannot verify from outside: you can audit your side of the boundary, not theirs, not their subprocessors, not an incident you would never hear about. That is not an accusation about any vendor. It is the shape of the arrangement.

Where the distinction stops being academic is professional obligation. Under the AICPA confidentiality rule, or Section 7216 for tax return information — the obligations we mapped for firms in How Accounting Firms Can Use AI on Client Data Without Exposing It — the disclosure to a third party is itself the event. What happens to the data afterward is not the trigger. The same logic runs through much of HIPAA and GDPR analysis, and through any cross-organization work where two parties cannot expose data to each other regardless of what a vendor promises either of them.

For many workloads, zero data retention is enough. The question is whether your obligation concerns what a provider keeps, or what leaves your control.

The third variable: who decides

Pratap Tambe added something the two-peril split does not capture. Decision authority.

An agent that generates a list of justified options, filtered by automated and manual checks before a human decides, has a different loss profile than an agent that decides and acts. Same model, same non-determinism, different exposure to consequence. His example was agentic AI producing inputs with a full audit trail into a human underwriter's decision.

This matters for insurability because decision authority is demonstrable. You can show where the human sat in the chain. Advisory and authoritative systems should not be underwritten as the same product.

Why classification comes first

Avinash Chauhan made the point that organizes all of this: before asking whether AI is insurable, ask which AI risk is being discussed, because that question alone changes the underwriting conversation.

Barry's answer to him was the cleanest statement of it. AI is an umbrella term covering a portfolio of technologies, each with its own portfolio of applications. Some are deterministic and insurable. Some are not. Treating "AI" as one homogeneous thing guarantees a bad answer, whether you are writing a policy, buying coverage, or setting internal governance.

Designed, deployed, and the gap between them

Vinay Bansal raised the objection that any honest version of this argument has to answer. An exposure bound describes the system you designed. The system that matters at the moment of loss is the one actually running. Same code, unmodified, boundary enforced, no path quietly routing values somewhere they should not go.

He is right that the gap exists. Where I land differently is on what the gap requires.

If the deployed system must be trusted with plaintext, then yes, you have to attest the runtime, because the runtime is the thing standing between the data and everyone else. But if the data is encrypted before it leaves the customer and the keys never leave the customer, a deployed system that has drifted from its design still cannot produce plaintext. There is nothing there to read. What has to hold is key custody on the customer's side, not the integrity of the processing environment.

That does not make verification pointless. It changes what verification is for. The useful artifact is compliance evidence: proof that a specific analysis ran against a specific committed dataset, checkable later by someone who was not in the room. That is where our own work points, and it is a different job from establishing trust in an execution environment.

Zero trust before verified trust. Reduce what has to be trusted first, then build machinery to verify whatever is left.

The lesson from a market that should not exist

James Clark Mitchell contributed the piece of history that reframes the whole question.

He once held a consulting contract to estimate the maximum possible loss from issuing commercial space permits to civilian companies. The loss was, as he put it, inestimable. They attempted it anyway. What created the market was not a better estimate. The US Department of Transportation agreed to assume liability for damages above 200 million dollars, and with the single-event loss capped at a number the satellite market could support, an industry was born.

That is the general pattern. Nuclear power got Price-Anderson. Terrorism coverage got TRIA after 2001. Commercial space got a federal backstop. In each case private markets could not size the tail, so someone capped it, and coverage followed.

Insurability has never required estimability alone. It required a boundable loss, and the boundary had to come from somewhere.

Which leaves AI in a particular position. There are two ways to cap a loss: statute or architecture. No one is capping AI liability by statute today, and it is reasonable to expect none until after a correlated loss event large enough to force it. That leaves architecture.

And architecture can only cap one of the two perils. It cannot cap what a model says. It can cap what a model sees.

What this means in practice

Separate the two perils in your risk register. They have different owners, different mitigations, and different residual risk. Managing both with one acceptable-use policy is the common mistake, and it is why so many AI governance programs feel heavy and ineffective at the same time.

Treat output risk as a scope and authority problem. Decide which decisions can tolerate a probabilistic input. Put human authority where the consequence is severe, and make the checkpoint real rather than ceremonial.

Treat exposure risk as an architecture problem. Ask one question of every AI system you run: what could this system read, retain, or transmit if everything went wrong at once? If the answer depends on a vendor's promise, the risk stays contractual and the boundary moves whenever the contract does.

Expect to be asked for evidence. Bank examiners have begun asking what AI systems can access. Insurers are writing AI exclusions and will eventually write questionnaires. In 2026 a US bank filed the first SEC disclosure attributing a data incident to an employee using an unauthorized AI application. The direction of travel is from stated policy toward demonstrated control.

For insurers and brokers, the same split suggests where coverage can move first. Exposure risk has a describable failure surface, a boundable maximum loss, and controls that can be checked at execution rather than attested on a form. Output risk may remain excluded for years.

Barry would add, and I would not argue, that no one is obligated to fix that. But uninsurable and unmanaged are not the same thing. If coverage keeps shrinking, what remains is reducing what the technology can reach in the first place.

Nera Systems builds infrastructure that lets regulated enterprises run frontier AI on their most sensitive data without exposing it. The LLM gets the query. It never gets the data. Try it on your own data.

With thanks to Barry Rabkin, Pratap Tambe, Avinash Chauhan, James Clark Mitchell, David Wray, CJ East, and Vinay Bansal, whose public disagreement produced most of what is useful here. Several of them still disagree with parts of it.