PromptQL's Adam Malone says AI analytics projects fail less from weak models than from missing business context. He laid out six levels of AI adoption, four failed text-to-SQL and semantic layer projects, and why data professionals are becoming context providers.
Key takeaways
- Most organizations sit at level one. Personal AI productivity builds no organizational knowledge when people leave.
- Undocumented knowledge breaks AI analytics. Text-to-SQL and semantic layer projects stalled on knowledge that lives in people's heads.
- Context beats model size. Malone cited a UC Berkeley benchmark where shared context added about 7 percentage points to accuracy.
- The analyst's job becomes context provider. AI handles queries and reports; people supply tribal knowledge.
- Lock down data, open up knowledge. Agents should inherit the user's permissions at runtime, not hold their own keys.
Most enterprise AI projects for analytics don't stall because the model isn't smart enough, PromptQL's Adam Malone argued. They stall because the knowledge needed to answer real business questions lives in people's heads. In the Data Science Connect webinar "I Came From the Future: Your Job Doesn't Exist There," sponsored by PromptQL, Malone, Director of the AI Catalyst Team at PromptQL, described how that changes the work of analysts and data teams.
His version of the future, he said, is the optimistic one. "It's not scary, it's just different."
What are the levels of enterprise AI adoption?
Malone, who runs forward deployed engineering at PromptQL, walked through a six-level adoption scale. Level zero is AI theater: the CEO demos something at an all-hands. "Everyone claps. Nothing changes," he said.
Level one is personal productivity, where individuals work faster with AI tools. The problem, he said, is that when that person leaves, the knowledge captured between them and their tool leaves too. "You're just vibing with a tool." Level two is team workflows, which create what he called AI islands. Level three is organizational infrastructure, where non-engineers ship production workflows. Level four, where he placed PromptQL, is AI-native operations with shared context that compounds automatically. Level five, fully self-driving, no one has reached.
He said he expected most attendees' organizations to sit around level one, and the chat responses matched.
Why do text-to-SQL and semantic layer projects fail?
Malone described four customer cases from before they worked with PromptQL. A Fortune 500 security company spent 18 months on a natural language to SQL system and reached about 70% complete. Getting past that was the hardest part, he said, because what is never documented "is the knowledge that lives inside people's heads." Accuracy also collapsed on multi-table joins.
A Fortune 100 tech company spent about 12 months building an AI layer on its messaging platform, then found that "a chat tool isn't a context layer, it's a fire hose." A global financial services firm built a carefully governed semantic layer that was out of date the day it went live, because the business moved faster than its update process. And a mid-market e-commerce company's text-to-SQL copilot failed on the CFO's first question, margin by category, because three different margin definitions lived in different tables and no one had written them down.
What does research say about AI accuracy on real business data?
Malone cited a UC Berkeley benchmark of real-world data questions across multiple data sets and domains. He said the best frontier model at the time reached only about 38% accuracy. For companies with Sarbanes-Oxley obligations, he said, anything short of full accuracy isn't acceptable.
Most failures, he said, came from the model not understanding business context, not from picking the wrong data source. He listed four things the benchmark identified as hard: data spread across multiple databases, join keys that don't match across systems, unstructured text such as contract terms in notes fields, and domain knowledge, such as revenue meaning different things to the CFO and to sales. Shared context alone produced about a 7 percentage point gain, he said. "It's not more intelligent, it just has more real-world knowledge."
Will AI replace data analysts?
Query writing, data wrangling and rerunning last month's report are already handled by AI, Malone said. "It's not perfect, but it's good enough." What AI can't know on its own is tribal knowledge: which columns are deprecated, why customer tables were duplicated after an acquisition, who to ask about what.
His prediction is that the job becomes providing that context. "Acting as context providers is in my view what the one job will be in future," he said. He argued the work left for people, such as understanding customers' problems and negotiating changes, is the more interesting part.
In a demo, he showed PromptQL picking up an account staffing change from a team thread and adding it to a shared wiki, routing a front-end bug to a coding agent using context others had provided, and helping a non-technical marketer build a reusable framework for evaluating event sponsorships. Malone said the wiki, built from everyone's contributions, is the knowledge layer most organizations are missing.
How should AI agents handle permissions and guardrails?
Asked about guardrails, Malone separated data from knowledge. "Data is something that should be under lock and key," he said, citing pay, emails and health data, protected with controls such as row-level security or policy-based access. Organizational knowledge, such as how revenue is defined, should generally be open, with scoped exceptions for sensitive work like an acquisition.
He also said agents shouldn't hold their own credentials. "Permissions should be inherited and granted from the user" at runtime, he said, because giving an agent its own API keys breaks down once multiple people use it.
On models, Malone said PromptQL is bring-your-own-model and that enterprises are moving toward using the right model for each job, including open-weight and self-hosted models. Using top frontier models for simple tasks during a recent "token maxing" phase, he said, approached irresponsibility. The companies that win, he said, "won't be the ones with the best models," but those that capture and share context most effectively.
Continue the conversation
Data Science Connect's 2027 schedule includes 24 free live webinars across the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and Plug & Play Series. See the webinar schedule. Data and analytics leaders can also meet in person at COLLIDE 2027 on Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta.
Frequently asked questions
Will AI replace data analysts?
Malone says AI already handles query writing and routine reports, but analysts' work is shifting to supplying the business context AI lacks, not disappearing.
Why do text-to-SQL projects fail?
In Malone's examples, they failed on undocumented business knowledge, such as multiple margin definitions, and on joins across tables and systems.
What are the levels of AI adoption?
Malone described six: AI theater, personal productivity, team workflows, organizational infrastructure, AI-native operations and fully self-driving.
How should AI agents get data permissions?
Malone said agents should inherit permissions from the user at runtime rather than hold their own API keys, especially in multi-user systems.
Watch: I Came From the Future: How AI Is Replacing Traditional Business Intelligence on YouTube
Event: Webinars · More coverage: Webinars insights
