Why llm-d 1.0 Matters More to Enterprise AI Budgets Than Another Model
The open source llm-d project reached 1.0 this week with features built for AI agents. We argue that shared, open inference infrastructure gives enterprises leverage over the cost that will decide whether agents pay off.
By The AI Enterprise Magazine Team4 min readOpinion · Weekly briefing
Key takeaways
- Inference is the cost that scales. Agents make many model calls that re-read long context, so serving efficiency drives the bill.
- llm-d 1.0 targets agents. Session-aware routing, turn-priority scheduling and quality-of-service controls are built for agent traffic.
- Treat savings claims as claims. The 5 to 10 times cost reduction comes from IBM Research, a founding member, under specific conditions.
- Keep the option to move. Measure cost per task and ask providers whether they run open serving software.
The most important enterprise AI release this week was not a model. On October 20, the open source project llm-d reached version 1.0, a production-ready release of software for running large language model inference across a fleet of GPUs and other accelerators. It drew little attention next to earnings and chip news. We think enterprise leaders should pay closer attention, because inference is fast becoming the line item that decides whether AI agents pay for themselves, and open, shared infrastructure for it gives buyers leverage they have lacked.
What is llm-d, and what changed this week?
Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA founded llm-d in 2025, and it is now governed by the Cloud Native Computing Foundation, according to the 1.0 announcement published by IBM. The project says it has more than 500 contributors from over 25 organizations. It sits on top of model servers such as vLLM and SGLang and decides how requests are routed, cached and scheduled across hardware.
Version 1.0 adds features aimed squarely at agents: routing that keeps a session's context on the same servers, scheduling that prioritizes each turn of an agent conversation, and quality-of-service controls so latency-sensitive requests are not crowded out by bursty agent traffic. It also adds zero-downtime upgrades and a stable configuration API, the unglamorous work that operations teams need before they will trust software in production. "AI inference is quickly becoming a core cloud native workload," said Chris Aniszczyk, chief technology officer of the CNCF.
Adoption is real, if early. JetBrains said llm-d runs in production behind its Junie Lite coding plan, DigitalOcean uses it for serverless inference and managed agents, and Google Cloud is integrating it into Google Kubernetes Engine.
Why inference cost is the number that matters
Agents are expensive in a way chatbots were not. A single task can involve dozens of model calls, each one re-reading long context: instructions, tool definitions, documents and the conversation so far. Much of that work is repeated. Software that remembers what it has already processed and routes the next request accordingly can cut the bill sharply.
The llm-d project claims cost reductions of up to 5 to 10 times for context-heavy agentic workloads, citing IBM Research, and says IBM researchers saw a prefix cache hit rate of about 98% in one study. Those are vendor figures from a founding member, under specific test conditions, and they should be treated as claims until enterprises reproduce them on their own workloads. But the direction is right, and the mechanism is not mysterious.
AI Connect speakers have been warning about this cost for months. At The AI Enterprise Conference 2026, Verisk Global Chief Data Officer Louis Dimodugno cautioned that token prices change almost daily and could rise once frontier model providers go public. "The financial piece is the piece that's always going to get the most attention," he said, in a session we covered in our story on Verisk's self-funding AI strategy. An open inference layer is one of the few hedges an enterprise can build against that risk.
The case against running your own inference
The other side deserves a fair hearing. Most enterprises are not inference operators, and many should not try. Managed model APIs are simple, scale on demand and shift the burden of GPU scarcity to someone else. Running a distributed inference fleet means hiring engineers who understand caching, parallelism and accelerator quirks. "Hosting your own models is a really big investment," Alteryx principal engineer Jonathan Lounsbury said in a AI Connect webinar, in our coverage of self-hosted LLMs and lock-in. His advice was to prove use cases with major providers first.
That is sound advice, and it does not contradict our argument. The point of llm-d is not that every company should run GPUs. It is that the economics of inference should not be a black box controlled by a handful of vendors. A common, open serving layer that cloud providers, neoclouds and enterprises all use gives buyers a way to compare prices, move workloads and negotiate.
What enterprise leaders should do now
First, measure inference cost per task, not per token. Agents multiply calls, and the only number that matters to a CFO is what a completed workflow costs against what it returns.
Second, ask your providers how they serve your traffic. Do they reuse cached context across agent turns? Do they run open serving software you could run elsewhere if prices change? The answers say a lot about future pricing power.
Third, keep the option to move. Dave Mariani of AtScale argued in a AI Connect panel that AI infrastructure must be open because "we don't know where this stuff is going," as we reported in our story on semantic layers, GPUs and inference costs. Fulcrum Digital founder Rajesh Sinha made a similar point about oversight at The AI Enterprise Conference 2026, in a talk we covered on sovereign AI architecture: "Have your own control panel, which is vendor agnostic."
Model launches will keep grabbing headlines. The enterprises that turn agents into durable returns will be the ones that treat the cost of running those models as an engineering problem they can influence, not a price they simply accept.
Frequently asked questions
What is llm-d?
An open source project, governed by the Cloud Native Computing Foundation, that routes, caches and schedules large language model inference across a fleet of accelerators on Kubernetes, working with model servers such as vLLM and SGLang.
Who backs llm-d?
Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA founded it in 2025. The project reports more than 500 contributors from over 25 organizations.
Should every enterprise run its own inference?
No. Managed APIs are simpler for most teams. The value of an open serving layer is that it lets buyers compare providers, move workloads and negotiate on price.
Opinion pieces reflect the views of The AI Enterprise Magazine Team.
Filed under: AI Strategy & ROI, Data & AI Infrastructure, Agentic AI
