On an episode of The Enterprise AI Show about private GenAI stacks, Luke Marsden of HelixML explained why financial institutions kept telling him they couldn’t use OpenAI:
There’s lots of financial institutions who just can’t send their data to an OpenAI or an Anthropic because their security and compliance people say, we need to have three copies of everything we run in production so that we can cope with one of them going down. You can’t have an external API be one of those pieces.
That’s a resilience argument, not a privacy one. It sidesteps the entire debate about whether the vendor reads your prompts. The compliance objection isn’t “we don’t trust you.” It’s “you are a single point of failure we cannot operate, cannot inspect, and cannot fail over from,” and no amount of contractual assurance fixes that.
That argument generalises further than resilience. A dependency you cannot operate is also a dependency that can be acted on without you. Whoever has the legal power to force that provider to hand over data reaches yours through a channel you don’t control and may never be told about. Resilience and compulsion turn out to be the same property seen from two sides: how much of your stack answers to someone who isn’t you.
Most sovereignty conversations start from privacy and end in a data residency map, the diagram showing which country the data sits in. Draw a boundary around Frankfurt, put the workload inside it, declare victory. The question a regulator, a board, or a serious auditor is actually asking is narrower and harder: who can legally force someone to hand this over, and how much can they force out? Geography is one input to that answer. It is nowhere near all of it.
%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#e8f4fd", "primaryBorderColor": "#3182ce", "fontSize": "14px", "fontFamily": "system-ui, -apple-system, sans-serif"}}}%%
flowchart TB
subgraph RES["What a residency map shows"]
F["Region: Frankfurt
data at rest, in the EU"]
end
subgraph COMP["What a compulsion boundary shows"]
P["US-parented group"] --> S["EU subsidiary"]
S --> O["Operating entity
holding the keys"]
L["Foreign court order
compelling access"] -.-> P
end
classDef ok fill:#e8f4fd,stroke:#3182ce,color:#1a202c,stroke-width:1.5px
classDef threat fill:#0f172a,stroke:#0f172a,color:#fff,stroke-width:1.5px
class F,P,S,O ok
class L threat
Both diagrams describe the same deployment. Only one of them answers the question you’ll be asked.
The regulation is messier than the claim
In the same episode, Marsden said something stronger: “all of European telco is regulated to not send their data to American tech companies, for example. The same is true for German healthcare.”
There’s no single law banning European telcos from using US providers. What exists is messier: national telecoms secrecy law, the ePrivacy Directive (the EU rule covering confidentiality of communications) sitting alongside GDPR, guidance from the regulators who supervise health and finance, and procurement rules that change from one member state to the next. On the ground the effect often looks like a ban. The legal basis is a patchwork.
The sovereignty conversation is full of confidently stated rules that turn out to be someone’s summary of their own risk committee’s decision. If you’re building the business case, cite the instrument. “Our regulator told us no” is a real constraint, but it’s a different argument from “the law forbids it,” and the second one falls apart in front of anyone who checks.
What actually changed in the last eighteen months
Here’s what makes 2026 different from 2024, when this was mostly a philosophical debate.
The EU Data Act started applying in September 2025, and Chapter VII is the piece nobody talks about. It requires cloud and data processing providers operating in the EU to put technical, legal, and organisational measures in place to stop non-EU governments getting at non-personal data held in the EU, where that access would be unlawful under EU or member state law. Before handing anything over, the provider has to judge whether the foreign request is reasoned, specific, and proportionate, and whether complying would break EU law or bypass an existing treaty for police and judicial cooperation.
Read that against the US CLOUD Act, the 2018 law letting US authorities demand data from US providers wherever in the world that data is stored, and the collision is obvious. This is the compulsion question written into EU law, and it covers non-personal data, which is most of what your AI pipeline actually moves. Everyone spent five years arguing about GDPR and personal data while the harder sovereignty question sat with the operational data nobody bothered to classify.
The legal basis for sending data to the US survived, but it’s under appeal. The EU’s General Court threw out Philippe Latombe’s attempt to strike down the EU-US Data Privacy Framework in September 2025, finding the US redress body independent enough and US bulk collection limited enough. He appealed to the Court of Justice, the EU’s top court, on 31 October 2025, case C-703/25 P. Still pending, no hearing date announced as of mid-2026.
So the framework holds today. If the Court of Justice rules against it, that’s the third transatlantic arrangement struck down in a row, after Safe Harbour and Privacy Shield. If your architecture assumes this one and has no plan B, you are one judgment away from an emergency migration, for the third time in a decade. At some point the pattern is the plan.
The European Data Protection Board, the body that coordinates the EU’s privacy regulators, raised the bar on calling a model “anonymous.” Opinion 28/2024 says a model counts as anonymous only if the chance of getting personal data back out of it, directly or by prompting it, is negligible for every single person whose data went into training. It also says that if the data was gathered unlawfully while the model was being built, using the model afterwards can be unlawful too, unless the model was properly anonymised.
That second point should worry anyone building on a model whose training data provenance is a shrug. You inherit the problem.
And the Commission is going further. In May 2026, CNBC reported it weighing rules that would restrict the use of US cloud platforms for processing sensitive government data across member states. Whatever lands, the direction isn’t subtle.
Meanwhile the AI Act’s rules for general-purpose models, the large foundation models everything else gets built on, have applied since August 2025, and the Digital Omnibus is now law. Regulation (EU) 2026/1744 was published in the Official Journal on 24 July 2026 and came into force on 27 July. The high-risk deadlines moved: standalone systems under Annex III, the hiring and credit-scoring and similar use cases, now 2 December 2027; systems built into regulated products under Annex I, medical devices and the like, now 2 August 2028. I’ve argued before that the delay is build time rather than a reprieve, and I’ll keep arguing it. The documentation and logging those rules demand are the same records you need to answer the sovereignty question anyway.
Nobody has mapped what actually leaves the boundary
Here’s the gap I run into on almost every engagement, and it’s the reason residency maps give false comfort.
Ask a team what data leaves their control when they call a hosted model. They’ll say “the prompt.” Then you start pulling the thread.
%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#e8f4fd", "primaryBorderColor": "#3182ce", "fontSize": "14px", "fontFamily": "system-ui, -apple-system, sans-serif"}}}%%
flowchart LR
U["One production
request"] --> A["The prompt
what everyone names"]
U --> B["Retrieved context
arbitrary chunks, chosen at runtime"]
U --> C["Embeddings
derived from your documents"]
U --> D["Tool call arguments
and tool results"]
U --> E["Traces and logs
shipped to a SaaS vendor"]
U --> F["System prompt
your business logic"]
classDef known fill:#e8f4fd,stroke:#3182ce,color:#1a202c,stroke-width:1.5px
classDef unknown fill:#0f172a,stroke:#0f172a,color:#fff,stroke-width:1.5px
class U,A known
class B,C,D,E,F unknown
Teams name the blue box. Everything dark crosses the boundary too, and most of it never makes it into the impact assessment.
Take those one at a time.
Retrieved context. If you’re doing retrieval-augmented generation, the system searches your documents and pastes the best-matching pieces into the prompt. Which pieces? Whichever ones score highest at the moment the question is asked. Nobody can write down in advance what might get sent.
Embeddings. If you call a hosted endpoint to turn text into vectors, those vectors are computed directly from your documents. They look like anonymous lists of numbers and they are considerably less anonymous than that appearance suggests.
Tool call arguments and tool results. For an agent, that means whatever it passed to your internal APIs and whatever came back.
Traces and logs. Your observability stack ships them to a SaaS vendor, and they contain all of the above.
The system prompt. It encodes how your business actually makes decisions, and it is often the most commercially sensitive thing in the whole system.
For agents this gets worse, because the list isn’t fixed when you design it. The agent picks which tools to call and what to retrieve while it’s running. You cannot produce a fixed inventory of what crosses the boundary, and a fixed inventory is exactly what a DPIA (the data protection impact assessment you file before deploying something risky) and an Article 30 record (GDPR’s required register of what you process and where it goes) both ask you for.
Same problem, different clothes: we assess once, up front, and then the system makes its real decisions later, while it’s running, where nobody is watching. Authorization has this problem. So does your data map. A list of what leaves your network, written at design time, describes an agent about as well as a cloud permissions policy describes what the agent will do with those permissions.
Sit down with a whiteboard and follow a single production request from end to end, naming every point where data leaves your network and the legal entity on the other end of it. The exercise regularly surprises the team doing it.
The stack is genuinely viable now
Running things yourself used to mean accepting a noticeably worse model. That gap has mostly closed.
Open weight models, the ones whose weights you can download and run on your own hardware, now trail the best closed models by months rather than generations. On everyday enterprise work the difference is a few percent, at somewhere between a quarter and a tenth of the cost. The licence matters as much as the capability: Qwen under Apache 2.0, DeepSeek and GLM under MIT. Your legal team will sign those off without a three-week review, which is not true of every model marketed as open.
For serving, Marsden’s advice from that episode has aged well: run vLLM, the open-source engine most teams end up on, on Kubernetes, and use the same deployment pattern in every region and environment instead of hand-rolling one per site. His warning about GPU scheduling still holds too, and it’s what bites teams first. “The Kubernetes primitives for requesting GPU memory are actually not very good. You can’t actually say, I want eight gigabytes of GPU memory, please.” Budget real engineering time for that. It is the least glamorous and most reliably underestimated part of the build.
Confidential computing covers the middle ground, where you want someone else to run the service but can’t accept an operator who is able to look inside. The hardware itself keeps the memory unreadable to the machine’s owner, and can produce a signed proof that it did, which is what people mean by attestation. Recent benchmarking on Blackwell shows roughly 98% of the throughput you’d get without it, and about a 1% hit at batch size 32 with 1024-token input and output. The overhead shrinks further as models get bigger, because compute and memory bandwidth dominate. “Too slow” no longer holds up.
And the big cloud providers have moved. AWS launched its European Sovereign Cloud in January 2026, starting in Germany with a stated €7.8 billion investment, 90-plus services, and a claim of no operational dependencies outside the EU. Microsoft’s EU Data Boundary has been adding AI services since mid-2025.
Treat those as a real improvement and an incomplete answer. The operator is still a subsidiary of a US-parented group, and whether that corporate structure actually defeats a CLOUD Act order is something lawyers are still arguing about, not something settled. That’s not a reason to dismiss it. It’s a reason to write the question down and get your own counsel’s view instead of treating a marketing page as a legal opinion.
How to actually decide
Sovereignty is a spectrum, and paying top-of-the-range prices for every workload is how these programmes die. Sort the work, then route it.
- Sort by who could be forced to hand it over, not by the sensitivity label on the folder. The question isn’t “how secret is this.” It’s “if a foreign authority served an order on our provider tomorrow, what would they walk away with, and would anyone tell us?” That question moves workloads around the matrix in ways the old labels never do.
- Trace what leaves before you buy anything. One production request, every hop, and the name of the company on the other end of each one. It costs an afternoon and it regularly changes the architecture people thought they were buying.
- Split the estate at least three ways. Self-hosted open weights for the workloads that cannot leave. Confidential inference with attestation where you want someone else to run it but need “we can’t look” rather than “we won’t look.” The public API for marketing copy and code comments, because paying the sovereignty premium on those is how you lose the budget for the workloads that need it.
- Write down what you’re assuming about the transatlantic transfer framework. If your design depends on it, say so in the architecture decision record, the document where you record why you built it this way, and note what you’d do if C-703/25 P goes the other way. One paragraph now, a quarter of panic saved later.
- Ask model providers where their training data came from, and mean it. Opinion 28/2024 makes their collection problem your deployment problem. A vendor who can’t describe how they sourced their data is selling you a liability with a chat interface attached.
- Use it to unlock the “no” pile. Somewhere in your organisation is a use case legal killed because the data couldn’t leave. That’s where the money is. Sovereignty work funded as a compliance cost gets cut. Funded as the thing that unblocks a dead project, it survives.
The takeaway
Residency tells you which country the bytes sit in. It doesn’t tell you who can reach them, under whose law, or whether you’d hear about it. That second question is the one that shows up in regulatory findings, and in 2026 it has hard legal edges it didn’t have two years ago: Chapter VII of the Data Act, a transfer framework under appeal for the third time, and a regulator position that makes someone else’s training data your compliance problem.
The good news is that the engineering excuse has expired. Open weights are close enough, vLLM on Kubernetes is boring and well understood, and confidential inference costs about one percent of your throughput. When someone tells you sovereign AI means shipping a worse product, that was true in 2024 and it isn’t now.
What’s left is the work nobody wants to do: tracing what actually leaves, naming the entity on the other end, and being honest about which of your rules are laws and which are your risk committee’s opinion. Both are real constraints. Only one of them survives contact with a regulator who asks for the citation.
References
Regulation
Transfers
Infrastructure
Podcast
- The Enterprise AI Show, “Building Private GenAI Stacks,” Luke Marsden (HelixML), for the three-copies resilience argument, vLLM on Kubernetes, and the GPU scheduling limits
