In a Data Science Connect expert panel, leaders from AtScale and Rhino Federated Computing laid out what production AI infrastructure needs, why prompt engineering won't fix accuracy, and how to control GPU and inference costs as agents arrive.
Key takeaways
- Production AI has many layers. Compute, security and governance, a semantic layer, model registries, orchestration and interfaces all need planning.
- Prompt engineering doesn't scale. Mariani said piling instructions into a chatbot is whack-a-mole; data and governance must come first.
- Stay open. Semantic layers wedded to one BI tool or data platform are risky bets when no one knows where agents are heading.
- Split workloads to cut GPU spend. Run preprocessing on CPUs, reserve GPUs for inference and match latency to real business needs.
- Measure agent quality before scaling. Agents need limits, interruption points and observability, plus automated scoring of their answers.
Moving AI from prototype to production is as much an infrastructure problem as a modeling one, according to panelists at "AI Infrastructure That Scales: Foundations for Real World Deployment," an expert panel webinar from Data Science Connect sponsored by AtScale, Rhino Federated Computing, SAP and Udemy Business. Dave Mariani, co-founder and CTO of AtScale, and Rhino Federated Computing's vice president of customer solutions, introduced as Adish, took questions from moderator Erik Bauch, a vice president of data and analytics.
Bauch, who said he supports his company's CTO on these decisions, called 2026 the year "where rubber meets the road" for production AI.
What does production-ready AI infrastructure require?
The Rhino executive said customers want three things: reliability, speed from development to value, and scalability beyond a pilot on a laptop. To get there, he listed the layers a CTO or CIO should consider:
- Compute: GPUs are not automatically available, so capacity must be secured, whether on Kubernetes or rented from a cloud provider.
- Security and governance: policies covering who can access, download, write and evaluate models and agents. Unstructured data such as PDFs and images, once rarely accessed, now needs policies because agents and developers use it.
- AI-aligned data: a semantic layer, so agents understand the terminology used across the enterprise.
- Models: model cards and a registry so users know how a model was built and evaluated.
- Orchestration: coordinating multi-agent tasks and workflows from data preprocessing to inference and fine-tuning.
- Interfaces: an area "in flux," he said, including interfaces for human-in-the-loop review and observability.
Mariani argued that scalable infrastructure must be open. Semantic layers tied to a single BI tool or data platform are risky bets, he said, because "we don't know where this stuff is going." He pointed to open table formats such as Iceberg as a way customers avoid being locked into one engine.
Why can't you prompt engineer your way to production AI?
Mariani said the most common anti-pattern he sees is teams trying to fix accuracy by piling instructions into a chatbot workspace. He compared it to whack-a-mole: the business changes too fast, and every bad answer prompts another instruction.
Success, he said, comes from putting data and governance first. "Imagine agents at machine scale," he said. "They can do a lot of damage and they don't know any better."
He described a large customer that won approval for an AI project because its semantic layer already served 5,000 users in Power BI and Excel. That gave leaders comfort that governance was covered and answers were right.
How do you control the cost of AI inference?
The Rhino executive said many organizations underestimate inference economics. Inference is now part of ETL and everyday applications, so GPUs once used only for training are needed around the clock. GPUs are also not reliably available on demand, he said, which forces decisions about renting, reserving or buying.
His first cost lever is splitting workloads. Preprocessing and postprocessing are mostly CPU work, so Rhino packages each step in its own container and runs only inference on GPU machines. His second is matching infrastructure to service levels. Real-time fraud detection needs low latency, but a protein structure prediction "even it takes one minute or 10 minute nothing is going to change," so a queue and shared pool of machines can spread the cost.
Mariani said a semantic layer cuts cloud data costs in two ways. A caching layer avoids paying for repetitive queries, and machine-generated queries from a physical-to-logical map avoid inefficient ones. He said he has seen single malformed queries cost customers $10,000, and noted AI can issue far more queries than a human in the same time.
What changes in infrastructure when you move to AI agents?
Treat an agent like an ML model, the Rhino executive said, with a build, evaluate and deploy cycle. He also urged teams to separate predefined workflows, where resource needs are known, from agentic workflows that are created dynamically. Agents need limits and interruption points so one cannot consume every available GPU, and he called agent observability "a minimum requirement."
Mariani described how AtScale tests its MCP server: a regression suite scores answers from zero to one, so a drop signals something to investigate. Asked which observability tools he recommends, he said it is early and current tools are largely log based. The Rhino executive said his team is evaluating options including open-source tools such as Phoenix and LangChain tooling, and framed it as a build-versus-buy decision.
What advice did the panel leave for AI leaders?
Mariani's closing advice was to invest in business meaning and a context layer, because "only you know what your business meaning is." The Rhino executive's was simpler: "Don't throw out any data like PDFs, images. Use everything that you have and make them available for AI."
Looking ahead, the Rhino executive said he is watching new specialized inference chips, and Mariani said he is watching for the next major use case after code generation and for open-source agent frameworks.
Continue the conversation at DSC webinars
Data Science Connect's 2027 calendar includes 24 free live webinars in four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. In person, The AI Enterprise Conference returns on Tuesday, August 31, 2027, at Pier Sixty in New York City.
Frequently asked questions
What infrastructure do you need for production AI?
The panel named secured GPU compute, security and governance policies, a semantic layer, model cards and a registry, orchestration, and interfaces for observability and human review.
How can companies reduce AI inference costs?
Separate CPU-bound preprocessing and postprocessing from GPU inference, and match latency to the use case so non-urgent jobs can share queued compute.
How does a semantic layer reduce cloud data costs?
Mariani said its caching avoids repetitive queries and its machine-generated queries avoid inefficient ones that can run up large bills.
How do you measure the quality of AI agents?
AtScale scores its MCP server's answers from zero to one in a regression suite, so a drop in the score flags a problem to investigate.
Watch: AI Infrastructure That Scales: Foundations for Real World Deployment edited on YouTube
Event: Webinars · More coverage: Webinars insights
