Getting a model working in a notebook is easy; keeping it reliable in production is not. In a Data Science Connect webinar sponsored by Tecton, leaders from Tecton, Emory Healthcare and a BBDO agency explained how standardization, cross-training, unified frameworks and clear ownership speed the path to production AI.
Key takeaways
- Production means someone is on the hook. SLAs, alerts and ownership separate production AI from notebook prototypes, Stumpf said.
- Standardize, centralize, then automate. Stumpf's three pillars cut technical debt and, at Uber, made teams faster.
- Design to pivot. Emory Healthcare containerizes solutions so it can switch clouds or models, Sousa said.
- Avoid train-serve skew. Define each feature once in a unified framework and run it on both batch and streaming data.
- Measure from day one. Without measurable results, winning leadership support to scale is very difficult, Sousa said.
A data scientist can get a model working in a notebook in days. Keeping it running for real customers is a different job. "Once you go from prototype to production, things get serious," Tecton co-founder and CTO Kevin Stumpf said in the Data Science Connect expert panel webinar "Accelerating the Path from Data Science to Production AI." He was joined by Supriya Bhatia, VP and Head of Data Science and Advanced Analytics at a BBDO marketing agency, and Melissa Sousa, Corporate Director of Data & AI Strategy at Emory Healthcare. The webinar was sponsored by Tecton, and Brian Mink, co-founder and president of Data Science Connect, moderated.
What stops AI prototypes from reaching production?
Production systems come with service-level agreements, Stumpf said. If a fraud model or recommendation engine goes down, someone has to be alerted and on the hook to fix it, which a notebook on a laptop never requires.
Sousa, whose team had two generative AI solutions about to go live, said her challenges include a tiered support model and keeping RAG pipelines updated. The biggest is proving accuracy and value again right before launch. As use-case risk rises, and in healthcare lives can be at stake, "people are afraid to take their hands off the wheel," she said. Her answer is continuous monitoring of both ROI and accuracy, so teams can deploy slowly and then ramp up quickly.
Bhatia pointed to silos and gaps in infrastructure and data pipelines.
How do you reduce technical debt in AI projects?
Stumpf defined the symptom: a system with heavy technical debt is hard or impossible to change or debug. He described three pillars, in order:
- Standardization: version control, pull request reviews and common ways to measure data quality and model performance, so others can review or take over work
- Centralization: a central team of experts who review what goes into production and enforce those standards
- Automation: once systems are standardized and centralized, automate model rollout, feature pipelines and monitoring
He cited Michelangelo, the central machine learning platform he helped build at Uber, a company he said valued speed above everything else. Teams moved faster on the centralized, automated platform, he said. Skipping best practices may get a prototype out sooner, but "that's just the first one percent of the project." The other 99% is maintaining and handing it off.
Bhatia recommended reusable code, coding standards and strong version control. Sousa added documentation, which AI can now help automate, and "designing to pivot." Emory Healthcare containerizes its solutions, and requires vendors to do the same, so they can move between Azure and AWS or switch models if a vendor raises prices or a smaller model will do.
She also urged fixing data quality at the source. Data scientists often clean data just enough to make a proof of concept work. Choosing use cases with enough backing to fix source systems produces pipelines that are useful to the organization whether or not that model ships.
How should data science and engineering teams work together?
Sousa recommended cross-training so data scientists and data engineers understand how their shortcuts affect each other downstream. Stumpf agreed and advised investing in machine learning engineers, a hybrid role, even if people remain "90% data scientist but 10% engineer."
Bhatia said roles are merging. Where teams are too large to work as one, she recommended joint accountability for outcomes and "the famous c-word, communication."
What is train-serve skew and how do unified frameworks help?
Stumpf said one of the most common traps in predictive machine learning is feeding a model different data in production than it saw in training. His example: a fraud feature counting transactions on a card in the last 30 seconds. If the offline and online systems calculate it differently, the resulting skew hurts performance and is extremely hard to debug.
Unified frameworks such as Spark and Flink, which Tecton uses under the hood, let teams define a transformation once and run it against both historical and streaming data, he said. Sousa added a caution from her consulting days: a model built for a retailer's website was later meant for in-store tablets, and the team found the store database and purchase rules were different. Know up front where a unified data set is possible, she said.
How do you automate and govern production AI?
The gold standard, according to Stumpf, is managing the whole system as code in version control, with CI/CD pipelines that train, deploy and monitor models and roll back automatically when something looks wrong. Sousa said her team's harder problem is evaluating non-deterministic generative AI output. It uses Langfuse, an open-source LLM engineering platform, to trace prompts and results, code accuracy manually, and track latency and token use.
Sousa described five areas of governance: the roadmap, the organization, applications, the model life cycle, and, most important, the data. For reliability, Stumpf named observability and clear ownership. "If you have a diffusion of responsibility type problem, then you've got a huge problem as well," he said.
On balancing speed, cost and performance, Stumpf advised optimizing prototypes for speed and avoiding premature cost optimization. Sousa said it depends on the organization. Some, especially in science and healthcare, won't look at a model "unless it's 96% accurate all the time, every time." Her closing advice was to design for measurement from the start, because without it, winning leadership support to scale is very difficult.
Join the next conversation
Data Science Connect has 24 free live webinars on its 2027 schedule across the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. See the webinar schedule. Enterprise AI leaders can also meet in person at The AI Enterprise Conference on Tuesday, August 31, 2027, at Pier Sixty in New York City.
Frequently asked questions
Why do AI prototypes fail to reach production?
Panelists cited SLAs and support ownership, the need to re-prove accuracy and ROI before launch, siloed stakeholders and gaps in infrastructure and data pipelines.
How do you reduce technical debt in machine learning?
Stumpf recommended standardization, then centralization, then automation; Bhatia and Sousa added reusable code, version control, documentation and fixing data quality at the source.
What is train-serve skew?
A mismatch between how features are calculated for training and how they are calculated in production, which Stumpf said degrades model performance and is very hard to debug.
What does automated production ML look like?
Stumpf described the whole system managed as code in version control, with CI/CD pipelines that train, deploy, monitor and automatically roll back models.
Watch: Accelerating the Path from Data Science to Production AI on YouTube
Event: Webinars · More coverage: Webinars insights
