Real-time AI pays off only when fresh data changes the prediction. In a Data Science Connect webinar sponsored by Tecton, panelists from Tecton, Sam's Club and AT&T explained how to combine precomputed and streaming features, catch silent pipeline failures and cut the cost of real-time machine learning.
Key takeaways
- Real time isn't always needed. Fraud detection needs millisecond data; churn prediction can run nightly, Bleifer said.
- Mix precomputed and streaming features. Partial aggregations combined at request time balance freshness, latency and memory.
- Models fail silently. Monitor pipeline health and data quality so broken feeds are caught before model quality drops.
- Start simple on drift. Log production features and compare them with training data in hourly or daily jobs, then retrain.
- Reuse features and track consumers. Shared pipelines, lineage and cost tagging cut spending on duplicate or unused pipelines.
Real-time AI earns its cost only when fresh data changes the answer. That was a recurring point in "Architecting Real-Time AI Systems: Strategies for Handling Dynamic Data at Scale," a Data Science Connect expert panel webinar sponsored by Tecton. The panel included Matt Bleifer, group product manager at Tecton and former product manager for Twitter's machine learning platform; Manisha Goswami, senior director of data science at Sam's Club; and AT&T's director of advanced analytics. Brian Mink, co-founder and president of Data Science Connect, moderated.
When does AI need to run in real time?
Bleifer pointed to fraud detection, including account takeover, synthetic identity and card transaction fraud, along with real-time recommendations and insurance quotes based on what a user is entering. In fraud, he said, "it's not enough to be like what happened as of midnight last night." If an account is being hacked, the model has to see what is happening up to the millisecond.
Many use cases don't need it. Customer churn prediction can run nightly, he said, and some products are deliberately slow, such as a weekly playlist meant to be explored over several days. The decision depends on the product experience and on how much real-time data matters to what is being predicted.
The AT&T panelist, whose work focuses on customer experience, agreed that the choice comes down to how quickly a response is needed, because real-time infrastructure carries a cost.
How do you get fresh features without high latency?
Bleifer described two common approaches and their problems. Computing features at request time against an operational database can be slow, may not have years of history available, and puts analytic load on production systems. Moving to stream processors such as Spark Structured Streaming or Flink adds data engineering complexity and memory problems.
The approach he recommended mixes the two. Take a user's transaction count over five years: precompute the total up to midnight, maintain a running count of new transactions from a stream, and add the partial aggregations together at request time, with an online store in the middle. That balances freshness and latency while keeping the infrastructure memory footprint low, he said.
Goswami said scale starts with the data itself. At Sam's Club, member data covers both active and inactive members, and her team is working to keep only the data that truly needs real-time access in the fast-access layer.
How do you monitor real-time data pipelines?
Bleifer split monitoring into two camps. System-level checks cover job health, streaming throughput and data freshness. Data quality checks catch pipelines that drop to zero or start returning nulls. Those matter because "models tend to fail silently," he said. A broken pipeline may show up only later as falling model quality, unlike a software bug that stops a homepage from loading.
For drift, he recommended logging the features served in production and comparing their distributions against the training data with scheduled offline jobs. Day-level granularity is usually enough for drift. "In most cases start simple with these things, log the data and run like hourly or daily jobs on it," he said. When drift appears, the simple answer is to retrain, ideally through an automated pipeline.
Goswami added real-time alerts and fallbacks so systems don't need a person watching them, and said retraining should also happen on a fixed schedule regardless of drift. Her team is testing an internal chatbot for clubs and building checks for hallucination into the system before launch, not only after.
How do you balance real-time performance and cost?
Bleifer listed technical fixes such as caching expensive online features and bulk-loading tables to an online store rather than writing rows one at a time. The bigger savings, he said, are organizational:
- Feature reuse: Without the right structures and tools, separate teams keep recomputing the same features.
- Consumer tracking: Knowing who uses each pipeline reveals which features deliver value and which old pipelines are still running with no one reading them.
- Cost tagging: Tags down to the underlying infrastructure let teams calculate what each feature costs against the benefit it adds.
Goswami said to decide first which use cases truly need real-time responsiveness, then look at options such as knowledge distillation.
Which models run in real-time AI systems?
Most real-time ML is still classification, Bleifer said, such as fraud or click prediction, using logistic regression, decision trees, XGBoost, TensorFlow or PyTorch. Generative AI is starting to intersect with these pipelines. One Tecton customer feeds a support chatbot real-time context such as a user's recent tickets and activity. Others call an LLM inside the pipeline to extract entities or tone from text as it arrives.
On skills, Bleifer said teams need to span data science, data engineering and software engineering. Goswami said analysts will increasingly need to add business context and explain results. The AT&T panelist said early AI teams were treated as a special group, "we might as well have put lab coats on them," but sustaining AI "takes our entire village."
In the closing round, Bleifer urged investing early in ML infrastructure rather than bespoke pipelines, which he said "can help pay a lot of dividends." Goswami said to invest in data quality and architecture, because "whatever goes in, it comes out."
More real-time AI discussions ahead
Data Science Connect has 24 free live webinars on its 2027 schedule across the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. See the webinar schedule. For in-person sessions, COLLIDE 2027, the largest AI conference in the Southern U.S., brings more than 1,000 enterprise data and AI leaders to Atlanta's Sandy Springs Performing Arts Center on Wednesday, October 13, 2027.
Frequently asked questions
When does an AI system need real-time data?
When fresh data materially changes the prediction, as in fraud detection or live recommendations. Bleifer said use cases such as churn prediction can run on nightly batches.
How do you serve fresh features with low latency?
Bleifer recommended precomputing aggregates up to a cutoff, streaming new events since then, and combining the partial results at request time from an online store.
What should you do when a model drifts?
The panel's answer was to retrain, ideally with automated retraining pipelines, plus scheduled periodic retraining regardless of drift.
How can teams lower the cost of real-time ML?
Panelists cited feature reuse across teams, tracking which pipelines are actually consumed, infrastructure cost tagging, caching and model optimization.
Watch: Architecting Real-Time AI Systems: Strategies for Handling Dynamic Data at Scale on YouTube
Event: Webinars · More coverage: Webinars insights
