Every data leader may have an AI mandate, but many say their data isn't ready. In a Data Science Connect webinar sponsored by Monte Carlo, Shane Murray and Michael Segner explained the four places AI pipelines break, how AI agents can write monitors and trace root causes, and how to set quality levels by use case.
Key takeaways
- Data is the differentiator. Models are widely available; first-party data used to augment them sets companies apart, Segner said.
- Four things break. Data, systems, code and models can each cause bad AI outputs, and rules alone can't catch every anomaly.
- AI can write the monitors. Murray said Monte Carlo's monitoring agent cuts rule creation from weeks to minutes, with about 60% of suggested rules accepted.
- Match quality to the use case. Financial pipelines need penny-level precision; recommendation pipelines prioritize speed.
- Scale from experts outward. Start with internal tools reviewed by experts before releasing AI to customers.
Most enterprises have made AI a strategic priority, but far fewer believe their data is ready for it. In a Data Science Connect webinar sponsored by Monte Carlo, Shane Murray, Field CTO at Monte Carlo and a former senior vice president of data platform at The New York Times, and Michael Segner of Monte Carlo's product marketing team explained what AI readiness means in practice and why observability has to cover far more than the data itself.
Brian Mink, co-founder and president of Data Science Connect, introduced the session.
Why is AI-ready data a differentiator?
Powerful models are available to everyone, so they won't be what sets a company apart, Segner said. The advantage comes from first-party data used to augment and fine-tune models and agents. He compared an uninformed AI system to an intern shut in a closet with no access to systems or information: without context, it tends to make things up.
According to Segner, Monte Carlo's surveys of data leaders found that all of them had AI as a strategic priority and 91% were building an AI product, while 68% did not think their data was AI ready.
Murray said the shift has turned data reliability into a different kind of job. Data has moved from batch business intelligence reports to machine learning, customer-facing data applications and generative AI, all of which need consistent quality and uptime. "It's much more of an engineering problem than data was a decade ago," he said.
What are the four root causes of bad data in AI systems?
Murray described four places where data and AI pipelines break:
- Data: schema changes, nulls and incomplete records, plus new questions about unstructured data, such as whether it is representative of what users ask and whether it has gone stale.
- Systems: orchestrators and ETL tools such as Airflow and dbt, and now AI orchestration tools such as LangChain and LangGraph. "There are no systems that haven't failed at some point or at some scale," he said.
- Code: a bad SQL join in traditional pipelines, or in AI systems a bad prompt or a model update with unintended effects.
- Model: responses that aren't fit for purpose, low acceptance rates or hallucinations, even when data, code and systems are working.
Segner said the dependencies across these, such as different prompts behaving differently on different models, create compounding complexity. He cited an insurance company whose LLM rated a customer support call as "120% satisfied."
RAG pipelines, prompt engineering, fine-tuning and agent architectures add more data, code and models that can fail. "You can't really just write a rule to catch each type of anomaly," Murray said. Teams need observability across the whole system and the ability to pinpoint where to fix a problem.
How are AI agents being used for data quality monitoring?
Historical data quality approaches required stewards and engineers to write thousands of labor-intensive rules and still left spotty coverage, Murray said. Monte Carlo's monitoring agent recommends rules and thresholds by reviewing data, metadata and descriptions and analyzing data samples. Murray said that cuts monitor creation from weeks to minutes, and that customers accept about 60% of the AI-generated rules.
His example came from a Major League Baseball team with a table of every pitch in a season. LLMs created rules such as acceptable speed ranges for fastballs and curveballs, a validation that is hard to spell out as a business requirement.
For root-cause analysis, Murray described a troubleshooting agent, then in beta, that starts from one table and works recursively upstream, using many LLM calls in parallel to test hypotheses across data, systems, code and models. When he led data teams, he said, finding the root cause was the hardest part and often required several teams. Leading teams also track median time to detect and median time to resolve incidents on operational dashboards.
How reliable does data need to be for each AI use case?
The right level of quality depends on the use case, the speakers said. Financial pipelines may need to reconcile to within a penny of the general ledger, while content recommendation pipelines need speed but can tolerate less precision, Murray said.
Segner advised building data quality requirements to fit each use case. "You don't want to overbuild, because that's just overhead," he said. Operational response metrics, such as time to respond and time to fix, can be more standardized. Many teams tier their data, separating certified, high-priority assets from data used less often.
How should data teams prepare for generative AI?
Natural language query engines make data democratization harder, Murray said, because product managers, marketers and finance staff can now query data directly. Teams need to know which top-tier data is trustworthy enough to expose, keep metadata complete and define key terms such as page views or revenue so an LLM doesn't pick up conflicting definitions.
To scale beyond a single pilot, Murray said the most successful teams start with an internal tool used by experts who give feedback, then widen internal use, and only then go external. He advised building where a company is genuinely different, such as content and audience behavior for a media company. "Data teams especially today need to lean into what they differentiate," he said.
On evaluation, he said teams look at business metrics, acceptance rates of model responses and, almost always, some form of human evaluation.
Learn more in Data Science Connect webinars
Data quality and AI observability are recurring topics in Data Science Connect's free webinar program, which has 24 live sessions on the 2027 schedule across four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. Registration for 2027 is coming soon.
Frequently asked questions
What is data and AI observability?
As Monte Carlo's speakers described it, monitoring the data, systems, code and models behind AI applications to detect anomalies and trace problems back to their root cause.
What makes data AI ready?
Murray pointed to trusted, tiered data, complete metadata, clear semantic definitions of key business terms and monitoring that extends to unstructured data.
How do AI agents help with data quality?
A monitoring agent can recommend rules and thresholds by analyzing data and metadata, and a troubleshooting agent can test root-cause hypotheses across upstream tables in parallel.
How should teams evaluate generative AI outputs?
Murray said teams use business metrics, acceptance rates of model responses and, nearly always, some form of human evaluation.
Watch: Accelerating AI-Readiness: Scaling Quality With Data + AI Observability on YouTube
Event: Webinars · More coverage: Webinars insights
