In-Context Learning: Start With AI Safely, Vespa's Kristian Aune
In a COLLIDE 2024 session, Vespa.ai's Kristian Aune ran a classification experiment using retrieved examples and a language model, reaching 94% accuracy in half a day. He compared text and vector search, model sizes and example counts, and described a data flywheel.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Start simple. Aune said a simple classification task needs no machine learning team or big budget.
- Retrieval quality drives answers. The model is only as good as the examples it receives.
- Tune example count. In his test accuracy peaked at 10 examples, then fell.
- Hybrid search did not help here. Short plain-English queries gained nothing from adding text matching to vectors.
- Build a flywheel. Reusing model output and human review pushes accuracy higher.
At COLLIDE 2024, the COLLIDE Data & AI Conference in Atlanta in October 2024, Kristian Aune, Founder and Head of Customer Success at Vespa.ai, argued that companies can start using generative AI safely with a simple technique called in-context learning, and without a machine learning team. In a session sponsored by Vespa.ai, Aune ran a text classification experiment on his laptop and measured accuracy, cost and speed at each step. His best result, reached in about half a day of work, was 94%.
Why does AI give wrong answers when the input is wrong?
Aune opened with a restaurant search. He asked the AI search tool Perplexity for a dinner spot near him in Boston, and the first answer, a seafood restaurant, was a 10-hour drive away. When he pushed the tool, it recognized the problem as local and drew on Yelp, and the results were correct.
The lesson, he said, was that if you feed a model the wrong data, the results will be wrong. He also pointed to the risk of chatbots that nobody controls, citing the Air Canada case. Making retrieval high quality for a small, well-understood problem is essential to the user experience, he said.
What is in-context learning?
A foundation model is trained on broad data, Aune said, so it knows a lot but cannot know everything. A retrieval system supplies the missing facts. In-context learning means giving the model a few relevant examples with each request so it can make better decisions.
Aune stressed that you do not necessarily need a vector database, though a system with vector similarity features helps. His team's open-source code, he said, has been available for seven years, so attendees could repeat the experiment themselves.
How accurate was the experiment?
The test used an open data set of customer banking queries with about 70 categories. For each test query, Aune retrieved similar labeled examples from a search engine and passed them to a language model, which chose the category.
- Text matching baseline: plain keyword search predicted the right category in almost 80% of cases, and the language model corrected a further share of the wrong ones.
- Vector similarity: converting text into vectors, which Aune called "a long sequence of numbers," retrieved better examples and improved precision by about two points.
- Hybrid search: combining text and vectors made no difference, a result he called "very disappointing" given his company's belief in using multiple signals. He said the queries were short, plain English with no proper names or addresses, so text matching added nothing. The result stayed at about 92%.
How do model size and the number of examples affect results?
Aune ran Meta's Llama 3.1 8-billion-parameter model locally on his laptop. Smaller models failed more often, he said, and the smallest had a 30% failure rate, though its answers were close to the right category names. Using a hosted OpenAI model through its API gave similar or slightly better results.
More examples helped only up to a point. Accuracy peaked at 10 examples and then got worse, he said, because too much data seemed to confuse the model. With a paid API every extra token also costs money, so he advised finding the sweet spot for each use case. Giving the model 10 examples from different categories did not help on this data set.
Retrieval itself took about 21 milliseconds, Aune said, so the time and cost came from the model. He noted the trade-off between a faster, cheaper, less accurate model and a slower, more accurate one. Local models also require paying for the infrastructure to run them.
How can a data flywheel push accuracy higher?
To get past 94%, Aune said, teams should use the model's own predictions in later queries and have humans review a sample. He described three feedback cycles: the model's classifications stored and reused automatically, a human review that runs daily or as needed, and the use of the accumulated data to train ranking models, which a machine learning team would value.
"The data is super important," he said, urging teams to build test sets so they can repeat experiments. If the model's output is unusable, he added, systems can fall back to "unknown" or let the user choose a category.
His summary was that simple tasks work: "You don't need a machine learning team. You don't need a big budget." He said image-aware models were the next step, with a demonstration of a visual model answering a text question about a chart in a PDF. He predicted such models would reduce the need for OCR and extraction pipelines in the following year.
COLLIDE 2027 returns to Atlanta
COLLIDE 2027, the largest AI conference in the Southern U.S., will bring 1,000+ enterprise data and AI leaders to the Sandy Springs Performing Arts Center in Atlanta on Wednesday, October 13, 2027. Details are on the COLLIDE event page, and registration is coming soon.
Frequently asked questions
What is in-context learning?
Giving a language model a few relevant examples with each request so it can make better decisions.
Do you need a vector database for RAG?
Aune said no, though a system with vector similarity capabilities helps.
How many examples should you give an LLM?
In his experiment, 10 worked best, and more reduced accuracy. He advised testing for each use case.
How accurate was the experiment?
About 94% after half a day of work, according to Aune.
This article is based on remarks by Kristian Aune, Founder and Head of Customer Success, Vespa.ai at COLLIDE Data + AI Conference 2024 on October 10, 2024. More coverage: COLLIDE AI Conference.
Filed under: Data & AI Infrastructure, AI Use Cases & Applications
