Own, Rent or Pay per Token? Enterprises Rethink Where AI Inference Runs

For most of the past three years, the default enterprise answer to "where should our AI run?" was simple: wherever the model provider hosts it, paid by the token. This week showed how much that default is fraying. On September 1, Broadcom pitched a private AI stack built around controlling "tokenomics" and certified a new set of models for on-premises use. On September 2 and 3, two very different buyers, a quantitative trading firm and an engineering simulation platform, signed up with CoreWeave rather than building their own GPU clusters. And Dell reported a $95 billion AI server backlog that will shape what anyone can buy, and when.

Taken together, the week's news describes a market where large enterprises are no longer choosing between "the API" and "nothing." They are choosing among three options: rent tokens, rent infrastructure, or own it. Each comes with a different cost curve and a different set of risks.

Why are enterprises rethinking per-token AI pricing?

Token pricing was a gift during experimentation. Nobody had to buy hardware to test a chatbot. It becomes harder to manage when agents run continuously, call models many times per task and pull large amounts of context into each call.

Practitioners at AI Connect events have been blunt about it. Pushkar Rege, Vice President and Co-Founder of the Business Intelligence Program at Mastercard, told a panel at The AI Enterprise Conference 2026 that the cost model of software has changed, and "a wrong prompt could have you wake up the next morning with a really large bill." His teams now track cost per outcome, counting compute, AI fees and support, and work backward from the business result.

Vendors have noticed. At VMware Explore, Lenovo's Scott Patti said "token economics is emerging as a critical driver of adoption and ROI," according to StorageNewsletter's report on Broadcom's VMware AI Factory launch. AMD, a partner in that launch, stressed that running its GPUs on VMware's platform involves no per-token charges.

What does the "own it" option look like now?

The pitch for private AI has matured from "you can run models in your data center" to "you can run them like any other workload." Broadcom said VMware AI Factory cuts the time from bare-metal servers to a first served model from weeks to hours, and offers observability into token throughput, latency and GPU and memory use, so teams can see their own unit costs. It includes an AI gateway for routing prompts and rate-limiting token use, and sandboxes that isolate agent-generated code.

Model choice, once a weak point for on-premises AI, is widening too. Broadcom validated five models for VMware Cloud Foundation, including Nvidia's Nemotron 3, Google DeepMind's open-weight Gemma 4 and models from NEC, Alibaba Cloud and Z.ai, according to StorageNewsletter. The company said its default runtime supports more than 150 open models and cited its own survey finding that 56% of enterprises run or plan to run production inference on private cloud. That is a vendor survey from a vendor with an obvious interest in the answer, so treat it as directional.

The case for owning rests on more than price. Rajesh Sinha, founder of Fulcrum Digital, told The AI Enterprise Conference 2026 that enterprise AI has moved from an era of speed to one of control: "It's not speed. It's control." His clients, many of them banks operating across borders, need agents that run inside their own firewall to satisfy data residency rules.

Why do some firms rent GPUs instead?

Owning has a catch: the hardware has to be bought, powered, cooled and kept busy. That is where the week's CoreWeave deals come in.

Hudson River Trading, a quantitative trading firm, signed a multi-year agreement to run AI research and model training on CoreWeave clusters built on Nvidia's newest Vera Rubin NVL72 systems, according to the companies' announcement. HRT is not abandoning its own data centers. It will connect its on-premises environment to CoreWeave over dedicated high-bandwidth links. The reasons cited were fast access to the latest hardware, predictable performance as models and datasets grow, and operational support.

A day later, simulation platform Rescale said it would let its customers in aerospace, automotive, energy, life sciences and manufacturing run AI and simulation jobs on CoreWeave through its Kubernetes service, per StorageNewsletter. For an engineering team, that means access to specialized GPU capacity without running a cluster.

Both deals reflect a hybrid logic: keep sensitive data and steady workloads close, and rent bursts and frontier hardware.

Can enterprises even get the hardware?

Supply is the constraint that cuts across all three options. Dell said it booked $60.9 billion in AI server orders in its latest quarter and ended with a $95 billion backlog, according to its earnings release. Traditional server and networking revenue rose 122%, a sign that the buildout is pulling in ordinary data center gear as well.

An enterprise that decides to own its inference stack in 2026 is competing for components with AI clouds and model labs. One that rents from a specialist cloud is betting that provider secured supply first. Either way, the decision is partly a procurement decision.

Ownership also has a cost that does not show up in a vendor demo. GPUs are the most expensive part of the stack to buy, power and cool, and speakers at The AI Enterprise Conference 2026 warned that slow storage and poor scheduling can leave them idle. Utilization, not list price, decides whether owning pays off.

How should leaders decide?

The useful question is not cloud versus on-premises but which workloads belong where. A practical sorting:

• Rent tokens for spiky, low-volume or experimental use, and for frontier capabilities you cannot host.
• Rent infrastructure for bursts of training or research, or when you need the newest hardware before you could buy it.
• Own it for steady, high-volume inference on sensitive data, where you can keep GPUs busy and measure cost per outcome.

As AI Connect's coverage of the AI factory model has noted, the goal is time to value, not the size of the cluster. This week suggests the enterprises that get there will be the ones that know their unit costs well enough to place each workload deliberately.

Frequently asked questions

Is it cheaper to run AI inference on premises?

It can be for steady, high-volume workloads if GPUs stay busy, but idle hardware and supply-driven prices can erase the savings. Utilization matters more than list price.

Why do companies rent GPUs from specialist clouds like CoreWeave?

Firms such as Hudson River Trading cite fast access to the newest Nvidia hardware, predictable performance and operational support, while keeping sensitive work on their own systems.

What is token economics in enterprise AI?

It refers to managing the cost of AI by the tokens models process, including throughput, latency and cost per outcome, which grows more important as agents call models repeatedly.

Filed under: Data & AI Infrastructure, AI Strategy & ROI