AI Observability for LLMs: Fiddler's Krishna Gade on Trust Models

COLLIDE Data + AI Conference 2024AI Observability: The Key to Unlocking the Full Potential of Large Language ModelsWatch the full session · 27 min

At COLLIDE 2024, the COLLIDE Data & AI Conference in Atlanta in October 2024, Krishna Gade, Founder and CEO of Fiddler AI, argued that enterprises cannot unlock large language models without a new layer in the AI stack: AI observability. In a session sponsored by Fiddler AI, Gade said generative AI failures had already made some enterprises pause, and he described how his company scores LLM outputs for hallucinations, toxicity and security problems.

Why are enterprises cautious about generative AI?

Gade said the technology was advancing quickly, with generative AI driving a sharp rise in adoption over the previous year and a half. But he said it "can fail in mysterious ways" and hurt enterprise adoption.

His main example was the Air Canada chatbot. According to Gade, it hallucinated an answer about a refund policy for a passenger, and the result was a brand problem and litigation. He also cited a bot that was jailbroken into making racist comments.

For use in areas such as wealth management or customer service, he said, enterprises "really need to build trust." The concerns he listed included privacy leaks, since models can memorize training data, along with accuracy, bias and robustness.

What does regulation require for generative AI?

Gade said these risks affect a company's regulatory posture. He pointed to the EU AI Act and a California bill requiring generative AI products to disclose the source content behind their responses, which he likened to content attribution.

In the question period he added that regulations are high level. They do not prescribe thresholds, such as a hallucination score cutoff, he said. They expect a process for testing and monitoring models, and how a company implements it is up to the company. A sophisticated process with sophisticated tools is more likely to reassure regulators, he said.

What is AI observability?

Gade said it is not enough to have data ready for training and a platform for building models. Enterprises also need an observability layer, "almost like a watchdog," that continuously monitors the inputs and outputs of generative applications.

He grouped the metrics into three kinds:

• Hallucination metrics: whether an answer is coherent, relevant and faithful to the underlying source.
• Safety metrics: personal data detection, toxicity and jailbreak attempts.
• Operational metrics: latency, cost and return on investment, which he called important because "generative AI while it's all glamorous it's pretty expensive."

One customer example involved a marketplace that wanted its support team to get accurate summaries of conflicts between buyers and sellers. Gade said the summaries could not make things up.

How do trust models score LLM outputs?

Fiddler's approach, Gade said, is what it calls trust models. These are smaller background language models that act like "little humans," checking another model's output for hallucination or toxic behavior. He described the idea as LLM-as-a-judge.

Using a second large commercial model as the judge would roughly double generative AI costs, he said. Fiddler instead uses models on the scale of BERT, with 100 million to 200 million parameters, fine-tuned for the task. They are cheaper and fast enough to run as a guardrail, he said, adding a few hundred milliseconds if a team blocks bad responses before they reach a user. He said Fiddler partners with Nvidia so its metrics can be used with NeMo Guardrails.

In the question period Gade explained the hallucination check as a faithfulness score between zero and one that shows how likely a response is to come from the source data. The models are trained through distillation and supervised fine-tuning, and they are not perfect. He said they sometimes miss hallucinations and sometimes give false positives, and Fiddler provides benchmark data on its accuracy. The aim is a first line of defense so human reviewers can focus on the blind spots.

Why use an independent observability layer?

Asked why a company would not rely on its model vendor's own tools, Gade pointed to separation of duties, a "maker-checker" principle. A neutral third party, he said, can judge whether one model or another fits a use case, avoids lock-in, and works with the multiple models customers now run. He added that regulators are likely to want neutral dashboards too. He compared the product to Datadog for AI teams, deployable in a managed cloud or on premises.

Gade closed with four pillars of responsible AI: testing and evaluation, continuous monitoring, security, and governance. Companies that adopt generative AI with no controls take on risk, he said, while those that wait risk falling behind competitors.

COLLIDE 2027 returns to Atlanta

COLLIDE 2027, the largest AI conference in the Southern U.S., will bring 1,000+ enterprise data and AI leaders to the Sandy Springs Performing Arts Center in Atlanta on Wednesday, October 13, 2027. Details are on the COLLIDE event page, and registration is coming soon.

Frequently asked questions

What is AI observability?

A layer that continuously monitors the inputs and outputs of AI applications for accuracy, safety and cost.

What are trust models?

Fiddler's term for small fine-tuned language models that score another model's outputs for issues such as hallucination and toxicity.

Why use a third-party observability tool?

Gade cited a maker-checker separation, avoiding vendor lock-in and a neutral view for regulators.

Does regulation set hallucination thresholds?

Gade said no. Regulations call for a process to test and monitor models and leave implementation to the company.

This article is based on remarks by Krishna Gade, Founder & CEO, Fiddler AI at COLLIDE Data + AI Conference 2024 on October 10, 2024. More coverage: COLLIDE AI Conference.

Filed under: Observability & Data Quality, AI Governance & Trust