AI Hype to Real-World Impact: LLMs for Business Analytics
At COLLIDE 2023, AnswerRocket's COO and CTO discussed how to get past generative AI hype in business analytics. They argued for higher-order business questions, using models to plan analysis rather than write SQL, and starting with one recurring, labor-intensive task.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Ask the higher-order question. Finley said analytics teams should find out why the business is asking, then tie work to goals such as profitability.
- SQL is a step, not the solution. The model should understand the request, add business context and plan the analysis.
- Allow several right answers. Testing must tolerate variation but draw a hard line on hallucinations.
- Text is data. Reports, email and meeting notes can be used once the model is given the right text.
- Start with a Goldilocks use case. A recurring, effortful task with some thinking involved makes a good first automation.
At COLLIDE 2023 in Atlanta, AnswerRocket co-founder and COO Pete Reilly and CTO Mike Finley held a conversation-style session on how companies can move past the generative AI hype and get measurable value from large language models in analytics. In the roughly 22-minute talk, Reilly asked the questions and Finley, who said he has worked in AI for many years, did most of the answering.
Their message, as presented at the time: aim for business questions rather than faster queries, use the model to plan analysis rather than write SQL, and start with one recurring, labor-intensive task.
How do you get a 10x impact from LLMs in analytics?
Finley said many things can produce a 2x gain, such as applying a successful process to twice as many categories. A 10x gain, he argued, requires doing "something radically different" and hitting several of those 2x improvements at once.
The trap is what he called paving the cow path: doing the same work faster. Analytics teams should go back to the business, ask why people ask the questions they do, and tie the work to goals such as profitability, category expansion or channel improvements. An objective like delivering data on time, he said, is one generative AI is unlikely to help with.
Reilly said he sees the same pattern with customers who want to answer "what were sales by month" with an LLM. The higher-order question, he said, is how the business is doing against plan and its top-level KPIs.
Should an LLM just write the SQL?
Finley said people often ask whether AnswerRocket's Max product is "just writing SQL." He called that framing a mistake. If every question from a CEO had to be answered by a single SQL statement, the statement would be either enormous or too slow, and there would be no easy way to verify the results.
Instead, he said, the model's job is to understand what the user wants, add business context (which time period counts as growth, what the channels are), plan which data to pull and decide what analysis to run on it. A database query is one step in that chain, not the solution. The shift in development, he said, is toward the machine helping plan the analysis.
He added that models can also write the "last mile" code for formatting tables, PDFs and slides, so analysts can spend more time on new kinds of analysis.
How do you test a non-deterministic model?
Finley compared language models to people. They were trained on a huge volume of human language, he said, and like a colleague asked for "the last year of sales," they may reasonably pick different interpretations on different days.
That means a question can have several right answers, so testing has to allow for that. What teams must do is draw a hard line on hallucinations and make them detectable. He described a process of "using the machine to check itself," giving the model rules for what counts as correct.
What are the main risks and the unexpected benefits?
On the risk side, Finley listed unreliable infrastructure, variable answers and hallucinations. He compared model outages to early database outages and predicted models will become "cheaper, faster, more reliable" over time.
The benefit he highlighted was text. Because language models are trained to understand human language, businesses can now use what he called dark data: annual reports, email streams and meeting notes that are written once and never read again. "Text is data," he said, and treating it that way could itself be a 10x factor.
How does an LLM work with structured business data?
Finley explained that a model only understands text and has a limited context window, which he put at 16,000 to 64,000 "syllables" (his word for tokens). Dumping a million records into it does not work.
In Max, he said, the system identifies the information needed for a question, runs a query against a source such as SQL, DAX or an API, and turns the result into text. Feeding the model records like "Sally bought shoes on Thursday" yields little. Feeding it the output of an analysis module, the kind of result once handed to the business as a dashboard, lets it answer questions in useful ways.
Where should teams start?
Finley recommended a "Goldilocks" project: not so big it touches everything, not so small nobody cares. The right target is an existing problem that consumes recurring human effort, such as synthesizing information and preparing decks, and that involves some thinking.
Reilly added that teams promising to "put an LLM on top of all our data" will find nobody uses it. Departmental analysis done on a regular basis is where he saw enormous value. Finley closed by saying businesses that use these models will outcompete those that do not, while acknowledging ethics and privacy concerns.
Where to learn more
The AI Enterprise Conference 2027 takes place Tuesday, August 31, 2027, at Pier Sixty in New York City, and COLLIDE 2027 returns to Sandy Springs Performing Arts Center in Atlanta on Wednesday, October 13, 2027. Details are at /events/ai-enterprise-conference/ and /events/collide/. Registration is coming soon.
Frequently asked questions
Should an LLM just write SQL for business questions?
Finley said no. He argued the model's job is to understand the request, add business context and plan the analysis, with a database query as one step among several.
How do you test a non-deterministic language model?
Finley said teams should allow multiple right answers, draw a hard line on hallucinations and use the model to check itself against rules for what is correct.
What is a good first generative AI project?
Finley recommended an existing problem that takes recurring human effort and involves some thinking, such as synthesizing information and preparing decks.
What is the context window problem?
Finley said models can only take in limited text, so tools must select the needed information and convert query results into text before the model sees them.
This article is based on remarks by Pete Reilly, Co-founder and COO, AnswerRocket; Mike Finley, CTO, AnswerRocket at COLLIDE Data + AI Conference 2023 on October 3, 2023. More coverage: COLLIDE AI Conference.
Filed under: AI Use Cases & Applications, AI Strategy & ROI
