At COLLIDE 2025, Nanonets CEO Sarthak Jain shared what separates the few AI projects that succeed from the many that stall. Drawing on about 120 projects, he said success comes from long, hands-on iteration, human controls, clean data and bottom-up adoption, not budgets or executive mandates.
Key takeaways
- Plan for a long ramp. Jain said teams should start with a small share of automation and expect six to 18 months of hands-on iteration.
- Design for human intervention. Like autopilot, AI needs controls that let people take over when it fails.
- Fix the data before the app. Outdated knowledge bases produce nonsense answers, even from leading models.
- Bottom-up beats top-down. Budget, CXO involvement and the champion's technical skill showed no correlation with success, Jain said.
- Lead with a wow moment. Winning pilots use live demos, the right model for the job and goal-based iterations.
Reports that 95% of generative AI pilots fail should not stop companies from experimenting, Sarthak Jain, CEO and founder of Nanonets, told attendees at COLLIDE Data + AI Conference 2025 in Atlanta. In a session sponsored by Nanonets, Jain shared what his company has learned about the small group of AI projects that do succeed.
Jain said Nanonets serves 100,000 self-serve customers and has done more than 1,000 AI implementations. He said he wasn't sure the success rate is as low as 5%, "but it's definitely on the lower end."
Why keep investing in AI when most pilots fail?
Jain framed the moment as a classic innovator's dilemma. He pointed to BlackBerry, which kept improving its keyboard because its most demanding customers asked for it, while the disruption came from the touchscreen. Incumbents that only iterate on what already works, he said, risk being displaced.
He opened with a historical comparison: a 1903 New York Times article that estimated airplanes could take one million to 10 million years to develop, nine weeks before the Wright brothers flew. "We're kind of in that AI moment today," Jain said.
How do you define a successful AI project?
Nanonets looked at about 120 AI projects and studied the ones that worked, Jain said. Success meant at least one of three outcomes: positive end-user NPS, a reduction or reallocation of headcount, or lower turnaround times or error rates.
From those projects, he drew four lessons:
- Plan for a long ramp. "You really need to babysit the AI," Jain said. Teams should start with a small percentage of automation and plan a six- to 18-month journey before the AI can take over completely.
- Design for the human. He compared AI to airplane autopilot, which runs most of a flight while two trained pilots sit beside a cockpit full of controls. "Don't design the AI to take over. Design for human intervention," he said.
- Put an engineer on the team. Engineers and the people implementing the project should understand the business requirements and be invested in the outcome.
- Fix the data first. Jain described AI as garbage in, garbage out, and urged teams to spend time on the data feeding retrieval or training before they build the application.
Why does data quality matter so much for AI agents?
Jain used customer service as an example. It is where people have had the most success building agents, he said, but if the knowledge base is outdated, the agent will give nonsense answers.
He suggested a test anyone can try: ask ChatGPT to write code for OpenAI's own API. Because the models change so often, "even ChatGPT will hallucinate on OpenAI's own documentation," he said. The AI needs access to correct data at every point when it responds.
Do AI projects need executive sponsorship?
Not according to Jain's research. "The most successful projects are all bottoms up," he said. He pointed to code-writing assistants, which individual engineers adopted on their own. "You don't need a CIO to push an AI agenda," he said. Citizen developers who find uses for AI in their daily work are the ones solving the problems.
Several factors showed no correlation with success, Jain said:
- Whether the project sat in a cost center or a revenue center
- The technical expertise of the project's champion; a non-technical person can lead one
- Budget, since the cost of experimentation is low
- Formal proofs of concept or CXO involvement
Jain also pushed back on the idea that AI mainly replaces jobs. He argued that people leave jobs they find mundane, and that companies that don't bring in AI innovation risk losing them.
What does a successful AI pilot playbook look like?
Jain walked through a pilot with a customer he described as a several-hundred-billion-dollar company in the IT industry. The steps:
- Find a wow moment. "All of the AI demos that are winning have a wow moment that captures the imagination," he said. In this case it was automatic error correction: when a person fixed a data extraction mistake, the model learned from it in the next iteration.
- Show a live demo, not slides. The team demonstrated 10 sample documents flowing directly into SAP.
- Choose the right technology. Every model has different trade-offs in latency and accuracy. Here, the team used a small trainable language model because response time was critical.
- Iterate toward a goal. The team ran five iterations over three months to reach a high confidence score, which told users when to send a document to a person for manual review.
Are small language models better than large ones?
For converting unstructured data into structured data and automating a workflow, Jain said, Nanonets sees small, fine-tunable models work better than large language models. Retrieval is a good way to add data to large models, he said, but fine-tuning them per use case becomes cumbersome.
Answering audience questions, he said fine-tuning on a specific customer's use case is more beneficial than fine-tuning on an industry data set. He added that when Nanonets fine-tunes for a customer, it selects among about 200 different models and architectures based on the constraints of the application.
See more practitioners at COLLIDE 2027
Sessions like Jain's are part of COLLIDE, the largest AI conference in the Southern U.S., which brings together more than 1,000 enterprise data and AI leaders. COLLIDE 2027 takes place Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. Registration is coming soon.
Frequently asked questions
What makes an AI project successful?
In Nanonets' review of about 120 projects, successful ones started small, planned for six to 18 months of iteration, designed for human intervention, involved engineers and invested in data quality first.
Do AI projects need executive sponsorship to succeed?
Jain said formal proofs of concept and CXO involvement did not correlate with success; the most successful projects were bottom-up, led by people who use AI in their daily work.
How do you run a successful AI pilot?
Jain's playbook: create a wow moment, show a live demo instead of slides, pick the model with the right latency and accuracy trade-offs, and iterate toward a clear goal such as a confidence score.
Are small language models better than large language models?
For turning unstructured data into structured data and automating workflows, Jain said small, fine-tunable models work better, since fine-tuning large models per use case is cumbersome.
Watch: The 5% Club: What Successful AI Teams Do Differently on YouTube
Event: COLLIDE AI Conference · More coverage: COLLIDE AI Conference insights
