Most of the writing about AI agent risk is aimed at engineers. This one is for the person who signs off on the deployment, because the developments that should change your 2026 decisions are not technical at all. They are legal, financial, and organizational, and they have already happened.
Here is the short version. A working agent that impresses in a demo is not the thing you are buying. What you are buying is a system whose mistakes are now legally yours, whose value is far harder to prove than adoption numbers suggest, and whose failures show up on a regulator’s clock measured in hours. Three bodies of evidence make that concrete.
The courts already answered “who owns what the bot says”
The reference case is small in dollars and large in precedent. Air Canada’s support chatbot told a grieving customer he could apply for a bereavement fare retroactively. He could not; the policy did not exist. Air Canada argued, remarkably, that the chatbot was a separate entity responsible for its own statements. In Moffatt v. Air Canada, 2024 BCCRT 149, the British Columbia tribunal rejected that outright and held the airline liable for negligent misrepresentation, awarding $650.88 in damages. The tribunal member wrote the line worth remembering: it makes no difference whether the information comes from a static page or a chatbot (American Bar Association).
The damages are trivial. The principle is not: if your agent speaks to customers, its hallucinations are your legal commitments.
Then it got sharper. On May 21, 2025, US District Judge Anne C. Conway ruled that Character.AI’s app is a product under product liability law, subject to the same standards as a defective car or a contaminated drug. Strict liability, negligence, and wrongful-death claims all survived the motion to dismiss, and the First Amendment defense was rejected, with the judge stating she was not prepared to hold that words strung together by a large language model are protected speech (Akin Gump; Transparency Coalition).
Read those two rulings together and a litigation strategy comes into focus that has direct engineering consequences: plaintiffs increasingly draft complaints to target the architecture of the deployed system, its guardrails, defaults, and escalation pathways, so the case looks like a product-defect dispute rather than a content dispute. In plain terms: your escalation design is now a litigation artifact. The absence of a confidence threshold is a design decision a plaintiff’s expert will describe to a jury. A model should never be the authoritative source for your own policy, because that is a retrieval problem with a deterministic answer, and letting the model improvise it is choosing hallucination risk for no benefit.
Klarna, read correctly
The Klarna story is the most-cited cautionary tale in this space, and most retellings draw the wrong lesson from it.
The facts first. In February 2024, Klarna announced its OpenAI-built assistant had done the work of 700 agents in its first month, handling 2.3 million conversations, two-thirds of its chat volume, and cutting resolution time from 11 minutes to under 2 (Klarna; OpenAI). Fourteen months later, in May 2025, CEO Sebastian Siemiatkowski told Bloomberg the company had focused too much on cost and ended up with lower quality, and began rehiring humans into a blended model.
Two corrections that change the lesson. The headline “700 agents” was largely hiring avoided during a growth phase under a freeze, not 700 people laid off; and the widely quoted $40 million was a projected profit improvement, a forward-looking estimate, not an audited saving (Fast Company). More importantly, this was a scope correction, not a failure of AI at customer service. Klarna kept the assistant for routine volume and reintroduced humans for premium and complex cases.
The real failure was a metrics failure. Klarna measured success by average handle time and aggregate satisfaction, both of which stay healthy while a small, high-value cohort of complex cases, the disputes and hardship and fraud, quietly collapses. The missing capability has a name: confidence-aware escalation. Below one confidence threshold a query routes to a human; below a stricter one it escalates sooner; and when it does, the human receives the conversation, the agent’s reasoning, and any flags the system raised. Klarna’s 2024 deployment had none of that in production form.
The buyer’s takeaway: any team reporting agent success on a volume metric with no quality counterweight is reproducing Klarna’s 2024 dashboard. And there is a cost most replacement business cases never model. Reversing the layoffs, recruiting, onboarding, and retraining, is expensive. The true cost of full replacement includes the cost of unwinding it if it fails.
Most pilots never move a number
The pressure to deploy fast comes from a promised return that the data says usually does not arrive. MIT’s Project NANDA study, The GenAI Divide: State of AI in Business 2025, found that about 95% of enterprise generative AI pilots delivered no measurable P&L impact, against an estimated $30 to $40 billion of enterprise spend. The evidence base was 52 structured executive interviews, 153 survey responses from leaders, and analysis of more than 300 public AI initiatives (report PDF; coverage).
The exact percentage got contested, and fairly, because many of those pilots were never scoped to move P&L in the first place. But even the skeptical reading leaves you with most pilots stuck at pilot stage. And the authors are explicit that the divide is not driven by model quality. It is driven by approach. The winners shared a profile: tightly scoped, one clear pain point, a smart integration partner, and a fast path from pilot to production. Not one giant autonomous agent. A narrow one that worked.
That is the finding I would attach to every budget conversation, because it reframes scope as a control rather than a constraint. A narrow agent has a small capability set, which means a small attack surface, a small blast radius, and a clean business case that is actually measurable. “Do one thing well” and “be defensible” turn out to be the same instruction, arriving from the finance side and the security side at once.
The clock you are actually on
The last piece is timing, and it is the one that catches organizations flat. Regulatory notification windows for serious incidents are now measured in hours, not weeks. Financial-sector rules under DORA set a four-hour window for major incidents. NIS2 requires a 24-hour early warning. Newer frontier-AI incident-reporting regimes run on days, not months.
Set that against how most organizations actually run their agents. Survey after survey through 2026 finds the same structural gap: a large majority of enterprises keep discovering agents they did not know were running, only a small fraction have a formal process to decommission one, and self-reported visibility keeps rising while measured monitoring coverage stays flat. You cannot notify a regulator in four hours about a system you never inventoried. Discovery, not policy, is the binding constraint, and it is the cheapest thing to fix before it is the most expensive thing to explain.
The question that should reframe every deployment review is not “can the agent answer the question?” It is: can the institution prove the agent answered within policy, used approved data, escalated correctly, logged the decision path, and avoided unauthorized action? If you cannot answer that today, you are carrying a liability you have not priced.
The takeaway
None of this is an argument against deploying agents. It is an argument against deploying them the way the demo encourages. The courts have decided you own what your agent says. The Klarna arc shows that the metric that declares victory can be the same metric that hides the damage. The ROI data says the deployments that pay off are narrow and measured. And the regulatory clock says you need to be able to see and stop these systems long before an incident, not during one.
If you are deploying an agent that talks to customers or touches regulated data, the highest-leverage move is to get an honest read on your actual exposure before you scale it. That is what my free 30-minute assessment is for, and the agent security scorecard is a way to check where you stand in ten minutes. Either is cheaper than the unwind.
References
The Security Lab Newsletter
This post is the article. The newsletter is the lab.
Subscribers get what doesn't fit in a post: the full attack code with annotated results, the measurement methodology behind the numbers, and the week's thread — where I work through a technique or incident across several days of testing rather than a single draft. The RAG poisoning work, the MCP CVE analysis, the red-teaming patterns — all of it started as a newsletter thread before it became a post. One email per week. No sponsored content. Unsubscribe any time.
Join the lab — it's free Already subscribed? Browse the back-issues →
