AI pilots are easy; scaling them is not. In a Data Science Connect panel, leaders from AtScale, Monte Carlo and Revefi said trust is the dividing line, built through monitoring, semantic context and governance that works like brakes on a race car.
Key takeaways
- Trust is the gate to production. Teams need monitoring that shows AI outputs are safe and doing their job, Segner said.
- Semantic context makes answers repeatable. Passing metric definitions through MCP lets queries be deterministic, not probabilistic, according to Mariani.
- Adoption is the first KPI. If a tool is slow or clumsy, people try it once and never come back.
- Watch for siloed AI. Long-running pilots on their own pipelines need to graduate to a governed platform.
- Governance works like brakes. Guardrails plus change monitoring let teams swap models and prompts without freezing.
Getting an AI pilot working is the easy part. Running it across teams and workflows with consistent results is where most enterprises struggle, according to a Data Science Connect expert panel webinar sponsored by AtScale, Monte Carlo and Revefi. Moderated by DSC co-founder and president Brian Mink, "Scaling AI Across Your Business: Transform Processes, Secure Outcomes, and Empower Teams" brought together Dave Mariani, CTO and founder of AtScale; Michael Segner of Monte Carlo; and Girish Bhat, SVP at Revefi.
The panelists kept returning to one idea: AI scales when the people relying on it can trust what it produces.
Why do AI pilots stall before production?
Mariani pointed to an MIT study that found 95% of generative AI projects fail. Chatbots handle simple tasks well, he said, but go off the rails on complex ones without context and governance. He also cited what the study called a verification tax: checking agent output takes so much effort that it wipes out the expected ROI.
Bhat compared a 30-minute prototype built on a low-code app builder to a paper airplane next to the passenger jet an enterprise needs. Production requires repeatable, scalable and often automated workflows that integrate with pipelines built over years or decades, plus data with the right context, quality, observability and cost controls. Teams that adopt "a systems or a platform mindset" are the ones succeeding, he said.
Segner said the most successful agents he sees in Monte Carlo's customer base use data already inside a company's own product, where quality standards are high. His examples included an inventory and promotions assistant for auto dealers, a travel company recommending destinations from past user behavior, and forecasting recommendations in financial services.
How do you build trust in AI outputs at scale?
Segner described an architect so afraid of hallucinations that the team limited a technical-manual assistant to about 75 fixed question-and-answer pairs. The only way past that fear, he said, is monitoring that confirms outputs are both safe and doing what they're meant to do, at a cost that works at scale.
For structured data, Mariani said an LLM navigating tens of thousands of tables with no shared definition of a metric like gross margin has almost no chance of getting it right consistently. AtScale passes semantic definitions to the LLM through the Model Context Protocol (MCP), so its semantic engine generates the query. That makes the queries "deterministic, not probabilistic," Mariani said, with the same definition and the same query every time. Trust also means users never see data they aren't supposed to, he added.
What KPIs show whether AI is delivering value?
Answering an audience question, Bhat listed four kinds of metrics: business impact, adoption, reliability and performance, and governance and risk. Mariani and Segner both put adoption first. If a tool is slow or clumsy, people try it once and never return, Mariani said, so the rate of adoption is an easy-to-measure proxy for value.
Segner said Monte Carlo measures its own troubleshooting agent, which finds the root cause of data quality issues, mainly by adoption. He added a cautionary example. A pharmaceutical company used AI to enrich customer records, expecting fewer records to need human stewards. Instead, low-confidence records moved into the middle tier, and stewards saw more work. The project still worked and was widely used, but the team had to justify it on better data rather than reduced steward hours.
Is shadow AI the biggest risk, or is it siloed AI?
Mariani said the answer to shadow AI is building governance into the foundation so people can self-serve from trusted data. Otherwise, "people will find a way to do it on their own and you may not like what they do," he said.
Segner saw a different risk: siloed AI, pilots running on their own pipelines and databases that never graduate to a governed platform. Bhat invoked the "life finds a way" line from Jurassic Park and argued for incentives tied to outcomes. "It's not about eliminating shadow AI but rather about adopting AI for the right outcomes," he said. On protecting trade secrets, he said data leakage prevention has resurfaced, with about 70 tools now targeting the problem.
How can AI governance keep pace?
"You have to have brakes to be able to move quickly," Segner said, citing Formula 1. He described a media company afraid to swap embedding models for article tagging. Guardrails plus monitoring that flags the impact of model, prompt or data changes keep teams from freezing. "When you don't know what the outcome of a change is going to be, you're kind of frozen in place," he said.
Bhat favored continuous, decentralized governance over centralized committees. Mariani recommended securing data at rest with role-based, row-level and column-level controls, since no one can predict how an LLM will query it. In regulated industries, he said, financial services customers that already govern BI through a semantic layer are more comfortable putting generative AI on top of it.
For unifying predictive and generative AI, Segner recommended a single source of truth in the warehouse or lakehouse, and Mariani urged choosing the best tool for the job, since plain machine learning sometimes wins. Their closing advice: learn fast (Mariani), start with high-priority business outcomes (Bhat), and, in Segner's words, "Don't ship and hope. Ship and validate."
More enterprise AI panels on DSC webinars
Data Science Connect's 2027 schedule includes 24 free live webinars across four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. See the lineup on the webinars page.
Frequently asked questions
Why do generative AI pilots fail to scale?
The panel cited missing context and governance, a verification tax from checking outputs, poor integration with enterprise workflows, and data that isn't ready.
How do you measure the value of AI in production?
Girish Bhat tracks business impact, adoption, reliability and performance, and governance and risk; Dave Mariani and Michael Segner both treat adoption as the first proxy for value.
How does a semantic layer help LLMs answer data questions?
Mariani said passing semantic definitions to the LLM through MCP lets the semantic engine generate the query, so a metric like gross margin is defined and queried the same way every time.
What is siloed AI?
Segner's term for pilots that run on their own pipelines, architecture and databases for a long time without moving to a governed, monitored platform.
Watch: Scaling AI Across Your Business: Transform Processes, Secure Outcomes, and Empower Teams on YouTube
Event: Webinars · More coverage: Webinars insights
