An AI system can look perfectly healthy and still be wrong. In a Data Science Connect webinar, leaders from SAP, Dynatrace and RaceTrac explained what to measure beyond accuracy, how to catch drift, and how observability creates the audit trail AI governance depends on.
Key takeaways
- AI observability asks a new question. Beyond whether a system is running, teams must know whether its outputs are right.
- Look past accuracy. Groundedness, consistency, override rates, cost and business outcomes all signal whether AI is behaving as expected.
- Anchor drift detection to business cycles and use shadow deployments before cutting over to a new model.
- Make signals drive action. Degrading quality scores can trigger rollbacks or route individual outputs to human review.
- Tracing is the audit trail. Open standards such as OpenTelemetry capture each decision end to end for governance.
A production AI application can return a healthy response code in milliseconds and still be wrong. That gap is why AI observability is different from traditional monitoring, panelists said in the Data Science Connect webinar "Seeing Is Believing: Observability as the Backbone of Trustworthy AI," sponsored by SAP and Dynatrace.
Shawn Sanders, Director and Head of Enterprise Data Science at RaceTrac, moderated the discussion with Shubham Trivedi, Digital Solution Advisor at SAP, and Jason Ostrowski, Field CTO for North America at Dynatrace.
How is AI observability different from traditional monitoring?
Traditional observability answers whether a system is running, fast, available and error-free, Ostrowski said. "But then, AI has introduced a much harder question," he said. "Is it right?"
An AI system can hallucinate, give bad advice or create business risk while every system metric looks healthy. So AI observability has to cover decision quality, model behavior, output quality and business impact, not just performance, he said.
For Trivedi, who works between business processes and AI deployment, observability isn't a monitoring checkbox. It determines whether an enterprise trusts AI enough to let it touch real transactions in processes such as supply chain or HR.
What should teams measure besides accuracy?
Accuracy is only one piece, Ostrowski said. Teams also need to track retrieval effectiveness, safety, cost and business outcomes. Because AI systems are non-deterministic, simple pass-fail tests don't work. He described continuous evaluations, either written as code or run by an LLM acting as a judge, that score outputs for relevance, hallucinations, bias, grounding, completeness and policy compliance.
Trivedi named three signals he watches:
- Groundedness: whether an output can be traced to a real, verifiable data source
- Consistency: whether similar queries produce stable answers across users and time zones
- Override and rejection rates: when a human rejects an AI output, he called it "the most honest and a brutal signal" that accuracy needs work
To check accuracy on a new AI process, Trivedi suggested running it in parallel with the existing one. In an HR leave-request example, an employee could still raise a ticket while teams compare the AI's answer against the HR policy manual.
On cost, Ostrowski said token use barely mattered during pilots but becomes significant once workloads hit production volume. He recommended tracking input and output tokens, flagging spikes that may follow a model or system prompt change, and allocating costs by agent, team and business unit, much as FinOps does for cloud.
How do you detect drift in production AI?
Trivedi advised anchoring drift detection to business cycles, such as period-end and year-end close, rather than relying only on statistical thresholds, because what counts as normal varies across the calendar. He called shadow mode deployment, running a new model alongside production before cutover, one of the most reliable techniques he has seen, and said users often spot faults before formal metrics do.
Ostrowski said many enterprises track evaluation scores over time and run anomaly detection on them. A completeness score that slides from 100% is a signal to investigate recent changes or roll back.
How should teams act on observability signals?
Observability should drive action, not just dashboards and alerts, Ostrowski said. He described a customer that replaced manual PDF document processing with an AI agent. The team baselined quality and confidence scores, and when they degraded, it rolled back to the human process and reviewed the failed documents. That helped it improve the agent faster.
Trivedi added that for finance and procurement, where actions can't easily be reversed, low-confidence outputs should go to a human reviewer instead. He also called for proper ticketing, escalation and incident response, with feedback shared across business and data science teams.
How does observability support AI governance and audit trails?
The hard part is tracing everything an AI decision touched afterward, Trivedi said. That requires tagging each decision with its model version, inputs, confidence level and timestamp, ideally inside the same system of record as the business process.
Ostrowski pointed to OpenTelemetry as the industry standard for tracing an interaction from the user through prompts, retrieval, tool calls and database lookups, with OpenInference as another open-source option. That end-to-end trace provides the audit trail governance needs, he said.
Trivedi said governance "doesn't fail because the principles are wrong. It fails because no one connected the monitoring to the compliance requirement." Across many teams, he recommended shared metric names, logging schemas and thresholds, an approach he called guardrails as a service.
What's next for AI observability?
Both panelists expect AI observability to merge into broader enterprise observability. Trivedi said he is watching how observability extends to agent-to-agent behavior over the next 12 to 18 months. Ostrowski said teams are applying DevOps practices to AI: instrumenting from the start, running evals during validation, using progressive delivery and feeding production learnings back into development.
In the closing round, Trivedi said that without tracing a decision to its data, "you technically don't have an AI in production. You are having a confident guess." Ostrowski's version: "You can't trust what you can't see."
More webinars on trustworthy AI in 2027
Data Science Connect has 24 free live webinars on its 2027 schedule across four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. See the webinar schedule for upcoming sessions on observability, governance and agentic AI.
Frequently asked questions
What is AI observability?
Monitoring that covers not only whether an AI system is running but also its decision quality, model behavior, output quality and business impact, as Dynatrace's Jason Ostrowski described it.
How do you measure AI accuracy in production?
Panelists described continuous evaluations, including LLM-as-a-judge scoring, groundedness checks and running AI in parallel with an existing process to compare answers.
How do you detect drift in AI models?
Track evaluation scores over time with anomaly detection, anchor thresholds to business cycles, and test new models in shadow mode alongside production.
How does observability support AI governance?
End-to-end tracing records what an AI system saw, what it decided and why, which provides the audit trail needed for incident response and compliance.
Watch: Seeing Is Believing: AI Observability & Trustworthy AI with SAP & Dynatrace | Webinar Replay on YouTube
Event: Webinars · More coverage: Webinars insights
