Executive Data Science and Generative AI: Making Better Decisions
In a 2024 AI Connect webinar, leaders from Snowplow, SQream and Tecton discussed how generative AI complements machine learning in executive decision-making. They covered unstructured data, evaluation, ROI, KPIs and the build-versus-buy choice.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- ML still does the heavy lifting. Leven said structured enterprise data is still mostly an ML problem, and most ML work is data preparation.
- Unstructured data is the opening. Wang said language models can turn reviews and text into features for predictive models.
- Treat it like a junior analyst. Reid said executives can use generative AI to answer data questions faster and as a sounding board.
- Tie it to a KPI. The panel warned against AI for AI's sake and urged measurable goals and internal benchmarks.
- Pick the right model. A simple regression can beat an LLM on cost and speed; training a foundation model from scratch is rarely worth it.
In a 2024 AI Connect webinar, "Strategic Synergy: Elevating Business Decisions with Executive Data Science," three vendor-side practitioners discussed how generative AI fits alongside machine learning when executives make decisions. The panel was part of a webinar series on generative AI for executives and was sponsored by Snowplow, SQream, Tecton and one additional sponsor whose name the captions garble.
The speakers were John Reid, Senior Product Manager at Snowplow; Yaniv Leven, VP of Market Strategy at SQream; and David Wang, VP of Marketing at Tecton. Brian Mink, co-founder and president of AI Connect, moderated.
How can generative AI improve existing machine learning and analytics?
Leven started with a caution. In his experience, traditional machine learning is still the most commonly used AI in enterprises, and structured enterprise data cannot simply be fed into a generative model. The obstacles to ML are largely organizational, he said: building a model takes too long, data engineers and data scientists often report to different leaders, and "80% of ML modeling is actually data prepping." On the generative side, he said, quality and hallucination keep many projects from deployment, even with retrieval-augmented generation (RAG).
Wang said generative AI's biggest opening is unstructured data. Language models, he said, can turn product reviews into structured signals such as sentiment and the aspects customers like or dislike. Those can then become features and embeddings for churn or recommendation models. Tecton, he said, had announced a capability to abstract that process the day before the webinar.
Reid described generative AI as "like having a really good dedicated junior analyst." Executives can ask questions of their data that once sat in a queue for weeks, he said, and use the model as a sounding board for what the data says.
Is generative AI creating new value or just efficiency?
Wang argued for net new value. Because general-purpose models are already pretrained, he said, companies no longer have to train everything themselves. The opportunity is real-time personalization, like a travel agent who knows your preferences, rather than a chatbot that answers "Paris" for everyone.
Reid gave an example from a customer, a vacation home provider, that precomputed features from listing photos so users could search for something as specific as a full-length mirror. He advised starting with simpler internal use cases, where quality can be evaluated before anything faces customers.
Leven said generative AI excels where the data is complex text, such as checking hundreds of pages of policies. That work is hard to scale with people, he said, and human reviewers rarely interpret documents consistently.
Can generative AI understand structured enterprise data?
Leven called this the largest untapped market and said it is not yet available. LLMs can report a single figure, he said, but cannot reliably explain why sales are declining. He doubted that models can infer the meaning of columns in real systems, citing a manufacturing customer with two million columns and CRM schemas with tens of thousands. Understanding a schema, he said, is "a demo level" capability.
Reid countered that data catalogs, knowledge graphs and metadata layers are advancing, and that language models remain useful on a subset of key metrics even if not on every column. Leven said he expected a mature solution in under a year.
How do you know when generative AI output is good enough?
Leven framed it as credibility, not just hallucination, since a model may pick one answer over a thousand conflicting sources. Where an error is strategic, he said, executives should hold off until credibility can be assessed.
Wang pointed to practices from traditional ML: CI/CD, feedback loops, testing against point-in-time accurate training sets and rich context. Reid recommended setting a measurable business goal and building an internal benchmark, a reference set of questions and answers, to check against human results.
How do you keep generative AI tied to business goals?
Reid warned to "beware of AI for AI's sake," noting that such projects are popular with executives and boards but go nowhere without a real business problem. Leven said disappointment comes from a gap between execution and expectation, so expectations should be set by anchoring the discussion in a specific business value.
Return on investment includes cost, Reid added, since running the latest models across millions of rows adds up and smaller local models can lower it. He said the technical team should decide the approach. When an executive asked about using OpenAI for time-series forecasting, his team replied that a regression model would be faster and cheaper.
Wang added that KPIs must be technical as well as strategic. "The last thing we want to do is put a cool chatbot onto our website and have zero conversion," he said.
Build, buy or customize?
Reid said training a foundation model from scratch is "almost certainly not worth" it, while fine-tuning, added context and RAG customize models without that cost. He suggested hosting an open-source model such as Meta's Llama as a home-grown route, and noted that latency can rule out a hosted API for real-time use. Leven said teams should first look at how the data behaves and pick the model family accordingly. Wang advised weighing vendor lock-in and keeping flexibility across models and data platforms.
Keep learning with AI Connect
Executives weighing these trade-offs can meet peers at The AI Enterprise Conference on Tuesday, August 31, 2027, at Pier Sixty in New York City. AI Connect also offers 24 free live webinars on its 2027 schedule; see the webinar page. Registration is coming soon.
Frequently asked questions
Should executives use generative AI or traditional machine learning?
The panel said it depends on the data and task. Generative AI suits complex text; for structured data and forecasting, traditional ML is often faster and cheaper.
How can generative AI improve machine learning models?
Wang said LLMs can convert unstructured data such as product reviews into structured features and embeddings that feed churn or recommendation models.
How do you judge whether generative AI output is good enough?
Reid recommended building an internal benchmark of questions and answers and comparing results with humans; Wang pointed to CI/CD and feedback loops.
Should companies train their own LLM?
Reid said training a foundation model from scratch is almost certainly not worth it; customizing with fine-tuning, context or RAG, or hosting an open-source model, are alternatives.
This article is based on remarks by John Reid, Senior Product Manager, Snowplow; Yaniv Leven, VP of Market Strategy, SQream; David Wang, VP of Marketing, Tecton; moderator Brian Mink, Co-Founder and President, AI Connect at AI Connect webinar, Generative AI for Data Executives Webinar Series (2024) on September 19, 2024. More coverage: Webinars.
Filed under: AI Strategy & ROI, Data & AI Infrastructure, AI Use Cases & Applications
