Two enterprises run the same open-weight model on the same afternoon. One pays per token through an API and trusts that the price holds. The other owns the GPUs, staffs the platform, and trusts that its own team can keep the thing upright.
Both are making an infrastructure bet with strategic consequences. Most leadership teams have not named it, and fewer have written it down.
The question is not cloud versus on-premises. That framing was obsolete before AI arrived. The real question is which workloads must stay under your control, which can safely rent capacity, and whether your architecture lets you change your mind later without rebuilding the application around it.
What enterprises actually report doing
The most cited recent figure comes from the Enterprise AI Infrastructure Survey 2026, commissioned by Cloudian and fielded in February 2026 through the Centiment research platform. It surveyed 203 enterprise IT decision-makers screened for authority over AI strategy or infrastructure.
Its headline findings: 93% of respondents had already repatriated some AI workloads from public cloud, were in the process, or were actively evaluating it; 79% had already moved workloads; and 73% planned further movement toward on-premises or hybrid infrastructure over the next two years.
Read the methodology before repeating those numbers. The sample is 203 people, it is self-reported, and it was commissioned by a vendor that sells on-premises storage. Two-thirds of respondents had AI in production, which skews toward organizations mature enough to have opinions about infrastructure at all.
What survives those caveats is directional and consistent with practitioner reporting: a meaningful share of enterprises have stopped treating public cloud as the default answer, and are making per-workload decisions instead.
The same survey identifies why. Ninety-one percent said that when deploying AI involving sensitive company data they would choose on-premises, private cloud or hybrid over public cloud. Fifty-eight percent said data residency concerns had directly delayed or scaled back an AI initiative. Forty percent reported actual cloud AI spending exceeding initial projections, and roughly half cited cost unpredictability as a barrier to expansion. Seventy-five percent identified current or planned workloads that require or would benefit from on-premises infrastructure for acceptable latency.
The case evidence is thinner, and more interesting
Surveys measure intention. Production deployments measure something else.
At Red Hat Summit in May 2026, BNP Paribas described a multi-year migration from hybrid cloud AI to entirely self-hosted models and infrastructure. The bank processes roughly 1.5 billion AI tokens daily, manages bare-metal clusters across three data centers for redundancy, and set out to make hardware behave like a service for more than 150,000 end users.
That is not a pilot. It is an operating model.
The most useful part of the account is what the bank's technical architect, Pascal Guerineau, said about cost. Total cost of ownership is lower than continuing with cloud-hosted AI, he said — but calculating the precise savings is not straightforward. Cloud GPU pricing is legible; in-house GPUs require counting server depreciation over years, data center costs, network costs, and staff.
That admission is worth more than any percentage. Most published self-hosting savings estimates quietly omit one of those four terms.
Other conference reports are narrower. Yapi Kredi described a 2025 move from a Cloudera-based MLOps system to a shared platform for predictive and generative AI, reporting 50% faster troubleshooting and 75% faster onboarding for data scientists. Northrop Grumman described provisioning services in its first on-site GPU farm, and said plainly that the cost comparison against cloud had not yet been calculated.
Those are vendor-conference presentations, not audited outcomes. They are still more specific than a survey response.
Why the cost argument is harder than it looks
The intuition driving repatriation is straightforward: steady, high-volume inference should be cheaper on hardware you own than on metered tokens. Sometimes it is. The claim is easy to overstate.
Three things complicate it.
Utilization. A GPU that idles is a depreciating asset producing nothing. Cloud pricing bundles elasticity, and elasticity has real value when demand is spiky or unproven. The economics flip only when utilization is high and predictable.
Model churn. Hardware bought for last year's model may be wrong-sized for this year's. The frontier moves; capability per dollar moves with it. Organizations that cannot refresh on a cadence can end up owning infrastructure optimized for a workload they no longer run.
Hidden operating labor. Monitoring, patching, lifecycle management, observability, power and cooling, and refresh planning are line items. The CNCF's July 2026 analysis of AI workload placement makes the point in a practical form: before moving serious AI workloads anywhere, organizations should run a readiness check covering accelerator capacity, storage performance, data locality, network isolation, identity integration, monitoring, backup, recovery, software supply, vulnerability management and policy enforcement.
The Omdia data reported from that same conference circuit suggests enterprises are responding rationally rather than ideologically. In a survey of 400 respondents conducted in October, nearly half were using open-source AI models. Asked how they reduce generative AI operating costs, 21% cited model efficiency techniques including quantization, and 18% cited running compute on-premises instead of in public cloud. Open-weight models as a cost lever came in at 4%.
That ordering is a useful corrective. The largest single lever reported is not relocation — it is making the model cheaper to run.
What quantization changed
The reason self-hosting is even discussable for mid-sized organizations is compression. As one practitioner analysis noted, 4-bit quantization techniques can shrink a 70-billion-parameter model from roughly 140 gigabytes of memory to something in the 30 to 40 gigabyte range, while retaining most of its quality. That puts a genuinely capable model on a single high-end workstation rather than a cluster.
The same analysis put an eight-GPU H200 server at roughly $250,000 — a figure cited by that author, not a quoted market price, and one that excludes the data center, the people and the refresh cycle around it.
The practical consequence is a portfolio. A frontier model for the hard reasoning work, a small open-weight model for the narrow, high-volume, well-defined tasks. Analysts quoted in the same reporting expect agent deployments to increase the value of smaller models, because a well-designed agent needs several models rather than one expensive one doing everything.
Three architectures, honestly compared
| Dimension | Cloud-only | Self-hosted | Hybrid |
|---|---|---|---|
| Time to first workload | Days | Quarters | Weeks to months |
| Capital exposure | Low | High, depreciating | Moderate |
| Marginal cost behavior | Scales with usage | Falls as utilization rises | Mixed by workload |
| Data residency control | Contractual | Physical | Physical for sensitive data |
| Model choice | Provider catalog | Whatever you can run | Both, by task |
| Failure mode | Price change, rate limit, deprecation | Skill gap, idle capacity, refresh lag | Fragmented controls and observability |
| Exit difficulty | Moderate to high | Hardware resale and staff | Depends on the abstraction layer |
Hybrid is where most enterprises land, and the reason is not compromise for its own sake. It lets a business separate where the data lives, where the model runs, and where the control plane sits. Common patterns include cloud for experimentation and on-premises for production, on-premises data with cloud model access over private connectivity, cloud training with local inference, and on-premises training with cloud burst capacity.
The cost of that flexibility is real: inconsistent controls, fragmented observability, and harder data lifecycle management. A hybrid estate that nobody has documented is not a strategy; it is an accumulation.
Sovereignty is more than where the server sits
Regulatory pressure rarely arrives as one clean rule. It arrives as a set of residency, audit and documentation obligations that differ by sector and jurisdiction, and it lands on AI programs as a series of questions nobody can answer quickly.
The CNCF analysis offers a workable definition. Since there is no single universally accepted definition of digital sovereignty, its authors treat it as an ongoing process built around five elements: operational autonomy, meaning control over and ability to manage all elements of a system; compliance, since without legal certainty no system operates sustainably; auditability, so that security and sovereignty are verifiable rather than asserted; portability, because technical, contractual or organizational dependency restricts autonomy and leaves a business exposed to sudden price increases; and resilience, since a system without redundancy and recovery is not sovereign.
That framing is more useful than a data center address. A workload can sit in a provider's region and fail the portability test badly. Another can run entirely on someone else's metal and still pass, if the application treats the model as a swappable component behind a stable interface.
The CNCF authors also expect AI pricing to rise, arguing that infrastructure investment at current scale cannot be sustained without revenue growth and that "just buy a subscription" thinking may end. That is a forecast from practitioners with a commercial interest in portability, and should be labeled as such. It is not a reason to repatriate; it is a reason to avoid a dead end.
The question that predicts your outcome
Practitioners debating this on the conference circuit in September 2026 did not converge on a winner. One analyst argued that faster GPUs create a gravitational pull toward centralization — the idea that agents on faster infrastructure simply make better decisions, faster, more often. Another argued that for steady, predictable, high-volume inference, ownership genuinely wins. A third reframed the decision as workload routing that balances quality, latency, data access, policy and cost.
All three landed on hybrid as the near-term reality. They disagreed on the tilt.
The variable that predicts who does well is not which architecture they pick. It is whether the application was designed so the choice can change.
That means a stable interface between the business logic and the model, so swapping providers is a configuration change rather than a rewrite. It means output constrained to a schema the application validates, so a different model does not silently alter downstream behavior. It means per-field clarity about which system is authoritative, because an agent reading two versions of the truth will reconcile them incorrectly and confidently. And it means evaluation harnesses that run the same test set against both paths, so the comparison is empirical rather than rhetorical.
Those are software engineering decisions, and they are made by whoever builds the application — long before anyone signs a GPU purchase order.
What to decide this quarter
- Classify every AI workload by sensitivity and latency. Regulated data and sub-second operational requirements go one way; drafting and summarization usually go another.
- Write the break-even method before the break-even number. Include compute, depreciation, power and cooling, data center space, network, platform staff and refresh. If your savings estimate omits one, it is a preference, not a business case.
- Test the exit. Take one workload and demonstrate that switching provider is a configuration change. Time it. If it takes a quarter, that is your dependency cost.
- Measure utilization before committing capital. Idle accelerators are the quietest way to lose money in this category.
- Keep the model portfolio decision reversible. Small open-weight models for narrow tasks, larger models for hard reasoning, with the routing logic owned by your application rather than a vendor's console.
Conclusion
The divide is real, but it is not between cloud and on-premises. It is between organizations whose AI capability is portable and organizations whose AI capability is rented.
A BNP Paribas can run 1.5 billion tokens a day on hardware it operates, and still admit it cannot cleanly quantify the savings. A mid-sized distributor should reach a different conclusion on economics and the same conclusion on architecture: keep the model swappable, keep the data classified, keep the evidence.
Control over AI infrastructure is becoming a competitive advantage for a boring reason. Not because owning GPUs is virtuous, but because the businesses that can move a workload when the price, the regulation or the model changes are the ones that are not renegotiating from a standing start. Everyone else gets to find out what their provider's next pricing page says.
Hashtags
#EnterpriseAI #AIInfrastructure #HybridCloud #SelfHostedAI #DataSovereignty #CloudComputing #AI Economics #AIGovernance #DataGovernance #CustomSoftware #SystemsIntegration #BusinessAutomation #EnterpriseArchitecture #DigitalTransformation #ITLeadership #CTO #CIO #RiskManagement #OperationalExcellence #CostOptimization #OpenWeightModels #ModelPortability #Kubernetes #Inference #VendorLockIn #BusinessStrategy #TechnologyStrategy #InfrastructureStrategy