In a Data Science Connect webinar, AtScale's Mark Palmer argued that enterprises get more AI value from better context than from bigger models. He shared benchmark results on semantic layers, the four most common sources of AI errors, and why teams should test AI accuracy every day.
Key takeaways
- Context drives accuracy. Palmer said adding a semantic layer lifted LLM accuracy on answerable benchmark questions from about 45% to between 83% and 94%.
- Four gaps cause most errors: vague prompts, vague semantics, bad data and questions the data can't answer.
- Don't let the LLM write the math. Palmer said LLM-generated SQL is the wrong path; a semantic layer should compute the answer.
- Test accuracy daily. A regression-style suite of questions and correct answers catches semantic drift when data changes.
- Curate, don't sprawl. A small, carefully designed semantic layer works better than thousands of auto-generated objects.
Enterprises looking for a better return on AI often reach first for bigger models and more compute. In the Data Science Connect webinar "Context Over Compute: The Overlooked Lever for AI ROI," Mark Palmer of AtScale argued that the bigger gains come from giving models the right business context. Muralidhara Lakshamanan, VP, Solution Architect at Synchrony, moderated the conversation.
Palmer, a former CEO of StreamBase and former chief product officer at TIBCO, now leads product and marketing strategy at AtScale. He said he would speak both as an industry analyst and as someone building context products.
Why does context matter more than model size for AI ROI?
Palmer disagreed with the idea that context is overlooked. He called it "really white hot right now" in the industry. The shift happens, he said, when companies move from using AI for creative work to connecting it to enterprise data.
"You cannot hallucinate your revenue numbers, your churn numbers, your metrics that drive your business," Palmer said. In his view, a knowledge base or ontology that describes the data is not enough. The system also has to handle the computation for the LLM, guided by a semantic layer and definitions of how metrics are calculated.
How much does a semantic layer improve LLM accuracy?
Palmer pointed to BIRD-Interact, an open-source benchmark that he said most major LLMs have attempted. He said average accuracy hovers around 22% to 23%, with the top of the leaderboard at about 38%.
AtScale spent four months benchmarking against it, according to Palmer. On answerable questions, he said, a model on its own reaches about 45% accuracy, and adding a semantic layer lifts that to between 83% and 94%. Gains ranged from 80% to, in one Gemini test, 345%, he said. Palmer noted that AtScale was still awaiting certified results.
He also said context can narrow the gap between models. In AtScale's testing, Kimi, an open-weight model, matched Sonnet's accuracy across a range of use cases when given a well-built semantic layer.
What context gaps cause AI errors in the enterprise?
Palmer listed four sources of error the benchmark helps identify:
- Vague prompts. He cited research attributing about 40% of AI inaccuracy in chat interfaces to vague prompts. A good system should stop and ask a clarifying question.
- Vague semantics. Customer Blue Yonder has 20 to 30 ways to report revenue, he said, and most companies have many.
- Bad or unclear data, such as encoded fields or columns without a clear join key.
- Unanswerable questions, where the data simply doesn't exist. LLMs tend to answer anyway, he said, and are often "very confidently wrong."
Ontologies and knowledge graphs give useful hints, Palmer said, and help with unstructured data. But they leave the LLM to compute answers probabilistically. "LLMs that generate SQL are wrong," he said. "It's just the wrong path."
How should teams fix an AI application that misses its ROI goals?
Measure it, Palmer said. He described building a test suite: list the 20 to 40 questions people commonly ask in a domain, usually finance first, and have data engineers write the correct SQL for each. Then check whether an LLM connected to the semantic layer gets the right answers.
The results show whether a problem comes from the prompt, the data or missing information, and what each correct answer costs. AtScale builds this kind of attestation framework into its product, which Palmer said it calls Canary.
He also urged companies to involve the analysts who used to build Power BI and Tableau dashboards. They know which version of revenue a user means and can build skills that steer the model to ask.
How do you keep AI context accurate as data changes?
Answering an audience question about stale context, Palmer said teams should rerun their accuracy tests every morning, like a software regression suite. If a question passed yesterday and fails today, a schema, metric definition or source document may have changed.
He warned against "semantic sprawl," where teams generate a semantic object for every table and end up with thousands that don't help. A smaller, carefully curated semantic layer mapped to real business questions works better, he said. Some large customers spend $30 million to $100 million a year on LLM consumption, and letting AI guess drives that cost up.
For a low-cost first step, Palmer suggested building a skill or plugin that tells the model not to guess revenue but to call a defined calculation. That helps "build determinism into a non-deterministic system," he said.
"It's not prompt engineering," Palmer said. "This is like semantic engineering."
Asked for his single lesson, he answered: "Measure it." Otherwise, he said, "guessing is not a strategy."
More on context and AI ROI in 2027
Data Science Connect has 24 free live webinars on its 2027 schedule across four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. See the full webinar schedule, or meet enterprise data and AI leaders in person at The AI Enterprise Conference on Tuesday, August 31, 2027, at Pier Sixty in New York City.
Frequently asked questions
What is context engineering for AI?
As Palmer described it, giving an LLM the business definitions and calculations it needs, often through a semantic layer, so it doesn't guess at enterprise data.
Does a semantic layer improve LLM accuracy?
In AtScale's benchmarking, Palmer said, accuracy on answerable questions rose from about 45% to between 83% and 94% when a semantic layer was added.
How do you keep AI context from going stale?
Rerun a suite of test questions with known correct answers every morning. A question that passed yesterday and fails today flags a change in the data or definitions.
What is semantic sprawl?
Palmer's term for generating a semantic object for every table, leaving thousands of definitions that don't help the model answer business questions.
Watch: Context Over Compute: The Overlooked Lever for AI ROI on YouTube
Event: Webinars · More coverage: Webinars insights
