At COLLIDE 2025, Snorkel AI Head of Solutions Dennis Panos argued that GenAI pilots stall because people don't trust them, not because models are weak. He laid out an evaluation-first process built with subject matter experts and said it took two stalled projects to production in about two months.
Key takeaways
- Adoption requires trust. Employees must believe a tool won't replace them and that its answers beat what they'd find themselves.
- Enterprise bars are higher. People forgive consumer chatbots, but expect enterprise AI to act like a capable colleague.
- Frontier models aren't enough. Each business must encode its own brand, policies and expertise on top of them.
- Evaluate first. Panos's process runs from product spec to expert-built evaluations, error modes, data development and fine-tuning.
- Stop working on vibes. Borrow software engineering's discipline of documented testing and iteration.
Most generative AI pilots that fail don't fail because the models are weak, Dennis Panos, Head of Solutions at Snorkel AI, told attendees at COLLIDE Data + AI Conference 2025 in Atlanta. In a keynote presented by Snorkel AI, Panos argued that trust is the real bottleneck between pilot and production, and that teams earn it by designing for the people who will use the system and testing against their standards.
Panos said he has spent a decade in enterprise AI, at McKinsey's QuantumBlack, DataRobot and now Snorkel, and one lesson has held: "If you design a system for humans to adopt it, you greatly increase the probability of being successful."
Why do GenAI pilots fail to reach production?
"For systems to be adopted, they must be trusted," Panos said, repeating the line for emphasis. In the enterprise, that trust has several dimensions. Employees need to believe a decision support tool won't replace their job, and that its answers are as good as or better than what they would find themselves or get from a colleague.
"When you don't build for trust you don't get adoption," he said. The stakes have risen, he added, because the innovation budgets that were plentiful from 2021 through 2024 are disappearing. The business is now paying for GenAI development, and it expects results.
Why is the bar higher for enterprise AI than for ChatGPT?
Panos described a paradox. People give consumer chatbots grace: experts using ChatGPT often spot answers they would have framed differently, but keep using it for what it does well. Enterprises expect much more. They want coding agents, decision support tools and customer service copilots to behave like highly capable colleagues.
He compared the situation to autonomous driving. Waymo and Tesla take very different technical approaches, he said, but both keep humans at the center because the risk of getting it wrong is so high. Practitioners, he said, should start from the end user's goal and business requirement rather than debating new protocols.
Why aren't frontier models enough on their own?
Frontier model providers are specializing, Panos said, moving from generalist annotators to specialists such as doctors and engineers to encode expert knowledge and reasoning. That may be the answer for some businesses, but not necessarily yours. "Your business is special," he said, with a brand built over decades, its own policies and procedures, and customers with different expectations.
GenAI, in his view, is "an extension of your workforce." He gave two examples: a decision support assistant that helps a credit officer compress a week of research into hours, and a customer-facing billing chatbot that must speak in the brand's voice and apply the right policies to a frustrated customer. Getting those policies wrong creates financial risk.
"You build trust one brick at a time, but you crash it down all at once," Panos said. Referring to a chatbot case involving an Air Canada customer, he argued that the effort to clean up such a failure, from the C-suite to legal and compliance, far outweighs the cost of building the system correctly the first time. Turning the temperature down to zero, he added, does not make a model deterministic.
What is evaluation-first GenAI development?
Panos urged teams to treat GenAI as AI software, managed from a product manager's point of view. His process:
- Write the product specification. Define what the agent or agentic system is supposed to do and its requirements for accuracy and referenceability.
- Build evaluations with the experts. The subject matter experts who will adopt the software should define the tests.
- Measure the foundation model. Use those evaluations to find gaps between the model's capabilities and the requirements, and identify error modes.
- Develop the data. Fill gaps with third-party data sets, expert annotation or sparing use of synthetic data, so as not to pollute the system.
- Fine-tune and repeat. Encode the knowledge into the system and run the cycle again.
The process is simple in principle, but "many enterprises that we work with are doing this on vibes," he said. ML teams, he argued, can learn from software engineering's discipline of documenting and testing, which lets teams roll back changes and iterate together.
What results did the approach produce?
Panos closed with two examples: 82% straight-through processing on incoming emails, with the remaining 18% flagged for human review, and 95% accuracy for a telecom billing chatbot representing a brand. Before Snorkel's involvement, he said, strong machine learning teams had worked on the projects for nine months and six months without meeting the business's accuracy requirements.
Applying the disciplined approach, from human-centered business requirements to product specifications, testing and error-mode development, took each project to production in about two months, Panos said. Building that way from the start, he said, helps end users adopt what teams build, speeds delivery and earns teams the right to do it again as AI moves from innovation budgets to business budgets.
Hear more enterprise AI leaders at COLLIDE 2027
COLLIDE, the largest AI conference in the Southern U.S., brings together more than 1,000 enterprise data and AI leaders. COLLIDE 2027 takes place Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. Registration is coming soon.
Frequently asked questions
Why do GenAI pilots fail to reach production?
Panos said the main barrier is trust: when systems aren't built for the people who must adopt them, they don't get used, and businesses now pay for GenAI development directly.
What is evaluation-first AI development?
An approach in which subject matter experts define evaluations from business requirements, teams measure foundation models against them, identify error modes, and fix gaps through data development and fine-tuning.
Does setting LLM temperature to zero stop hallucinations?
Panos said turning the temperature down to zero does not make a model deterministic; reliable systems need defense in depth and rigorous evaluation.
Why customize frontier models for the enterprise?
Panos said each business has its own brand, policies and customer expectations, so it must build specialized knowledge on top of improving frontier models to capture value.
Watch: Human-Centered Design in GenAI: Planning and Building to Cross the Pilot-to-Production Ravine on YouTube
Event: COLLIDE AI Conference · More coverage: COLLIDE AI Conference insights
