The biggest costs of production machine learning are often idle people, duplicated pipelines and models built before value is proven. In a Data Science Connect panel brought to you by Tecton, leaders from Tecton, Mosaic Health and JPMorgan Chase's Morgan Health explained how to scale ML while keeping costs in check.
Key takeaways
- People cost is the big cost. Stumpf said idle time from handoffs and over-specialization drives spend more than tooling does.
- Prove value before you build big. Kim has seen millions spent on projects with no business value.
- Avoid premature optimization. A quick production experiment can show impact before investing in back-end performance.
- Share pipelines and right-size compute. Compute and fast online storage dominate data processing costs; offline storage is now cheap.
- Measure, standardize, automate. Stumpf's closing advice for controlling AI and ML costs.
The most expensive part of production machine learning is often not the compute bill. It is idle people, duplicated pipelines and models built before anyone proved they were worth building. That was the message of a Data Science Connect expert panel webinar on the hidden costs of AI, brought to you by Tecton, with Kevin Stumpf, CTO and co-founder of Tecton; Suzanna Kim, director of data science at Mosaic Health; and Preeti Vaidya, vice president of analytics solutions at Morgan Health, part of JPMorgan Chase. Brian Mink, co-founder and president of Data Science Connect, moderated.
What changes when ML moves from notebook to production?
Vaidya said the biggest gap is a lack of upfront investment in data infrastructure that can support both experimentation and mature production data flows. Without it, a good idea built in a notebook can't scale in time, and a competitor gets there first. Just as important is the wider data ecosystem: teams, skills, data management, governance and quality. "We don't talk enough about the ecosystem, the boring stuff," Vaidya said, that has to happen for years before a polished product reaches the market.
Stumpf described three dimensions of scale. More people bring handoff and communication problems. More use cases lead teams to reinvent the wheel, such as rebuilding a feature pipeline that already exists. More data changes the architecture entirely: a fraud model that checks 100 deposits a day is different from one handling 100,000 transactions per second. "You're not going to hammer your Snowflake with 100,000 queries per second," he said.
Kim said data scientists used to working ad hoc in notebooks need to be told early that production code must be modular. Shared compute also needs coordination, since one large job kicked off midday can absorb the GPUs and CPUs other teams need.
What mistakes drive up the cost of AI and ML teams?
For Stumpf, organizational cost is ultimately people cost, and it rises when people sit idle waiting on other teams. Too much early specialization makes that worse, because shipping a model then requires five dependent groups. Companies that move quickly and cost-efficiently often hire machine learning engineers who span data science, data engineering and software engineering, he said. Where that isn't possible, automation such as pull requests that deploy automatically to staging lets people work self-serve.
Kim pointed to building too much too soon, citing projects where "millions have been spent and there was actually no business value." Teams should start small and verify impact before investing in production deployment and operational change. Kim also warned against rigid, monolithic code, where updating a single library can break everything.
Vaidya said the difference between well-adopted products and those that dwindle is buy-in from stakeholders, senior leaders and users from the start, often through a steering committee that defines the problem and validates ROI before heavy investment.
Should you optimize infrastructure before proving value?
Asked whether customer-facing AI can do harm before back-end performance is ready, the panel urged balance. Vaidya said models without MLOps scaffolding risk data drift and can be thrown off by events such as a policy change. Kim said slow, buggy or wrong responses cause customer friction, so there is a threshold to clear.
Stumpf warned against premature optimization. A quick, even inefficient, experiment in production can show whether a model drives bottom-line impact before a team invests in optimizing the back end.
Where do ML data processing costs come from?
Compute and fast online storage dominate, Stumpf said, while offline storage on services such as S3 has become cheap enough to almost ignore. He advised sharing pipeline outputs across teams, especially as LLMs make processing more expensive, and right-sizing CPUs and memory.
Kim cited siloed data scientists creating duplicate datasets, manually refreshing training data instead of automating with a tool such as Airflow, and starting from scratch instead of building on prior work. Vaidya named poor memory management, such as copying DataFrame after DataFrame, and code that doesn't take advantage of distributed infrastructure. Missing data dictionaries and quality checks also cost time.
All three agreed notebooks belong in experimentation, not production. Stumpf said running cells out of order makes results hard to reproduce, "a debugging nightmare."
How do you measure ROI for machine learning?
Stumpf said fraud detection shows clear value: companies have told him even a first ML prototype improved on rules-based systems by double-digit percentage points. Use cases such as personalization are harder to tie to the bottom line, so teams rely on holdout groups and proxies like customer satisfaction scores.
Kim described evaluating a model that selected members for palliative care programs, using propensity score matching and difference-in-differences regressions to measure cost savings shared with clients. A "digital nurse" tool that automated policy reviews of clinical documents saved 20% to 30% of manual review time, according to Kim. Vaidya said metrics should match the problem, such as care quality and adoption for tools meant to reduce physician burnout.
What are the cost implications of generative AI and agents?
Stumpf said model choice trades off speed, accuracy and cost. He recommended starting with a strong generalist model, then testing whether a smaller, distilled or self-hosted model offers better value, keeping in mind that self-hosting adds operational cost.
Vaidya warned against a proliferation of uncoordinated agents. "Just because a task can be solved by an AI agent doesn't mean it's a task that needs to be solved," Vaidya said. Kim cautioned against building too many custom solutions when paying for an existing API may be cheaper.
Stumpf's closing advice was to measure, standardize and automate. "You can't manage what you don't measure," he said.
More enterprise AI panels in 2027
Discussions like this one are part of Data Science Connect's free webinar program, which has 24 live sessions on the 2027 schedule across four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. Registration for 2027 is coming soon.
Frequently asked questions
What are the hidden costs of scaling machine learning?
The panel cited idle people waiting on other teams, duplicated pipelines and datasets, over-engineered projects without proven value, poor memory management and rigid code.
Should notebooks be used in production ML?
All three panelists said no in general: notebooks suit experimentation, but out-of-order cell execution makes production results hard to reproduce.
What drives ML data processing costs?
Stumpf said compute and fast online storage dominate, while offline storage such as S3 has become nearly negligible.
How do you measure ML ROI when impact is indirect?
Use holdout groups and proxies such as customer satisfaction scores, or rigorous methods like propensity score matching and difference-in-differences, as Kim described.
Watch: The Hidden Costs of AI: Optimizing Your ML Data Stack for Production Scale on YouTube
Event: Webinars · More coverage: Webinars insights
