Text extraction throws away the charts and tables in visually rich documents. In a Data Science Connect webinar sponsored by Vespa.ai, Vespa CEO Jon Bratseth and RavenPack's Aakarsh Ramchandani explained how ColPali retrieves PDF pages by what they look like, where it still falls short and how to scale it.
Key takeaways
- ColPali embeds image patches. A visual language model creates contextual embeddings for pieces of each page and matches them to text queries in the same vector space.
- It simplifies pipelines. RavenPack replaced a roughly seven-minute, multi-step OCR pipeline with retrieval in about a second, Ramchandani said.
- It doesn't replace OCR yet. Retrieval improves, but reasoning, parsing and verification are still needed on top.
- Binary vectors make scale affordable. Bratseth said binarized, shortened vectors give nearly the same results with less memory and CPU.
- Evaluate on your own data. Start with 10 or 20 PDFs from your domain and build evaluations before scaling.
Most retrieval-augmented generation (RAG) systems read a PDF by extracting its text, which throws away the charts, tables and layouts that often carry the most important information. In the Data Science Connect webinar "Transforming the Future of RAG with ColPali," Jon Bratseth, CEO of Vespa.ai, and Aakarsh Ramchandani, Chief Strategy Officer at RavenPack, explained how ColPali retrieves pages by what they look like, and where the approach still falls short. The webinar was sponsored by Vespa.ai, and Brian Mink, co-founder and president of Data Science Connect, moderated.
What is ColPali and how does it work?
Bratseth traced the idea through three steps. First came vector embeddings, which let systems retrieve text by meaning rather than exact words, though with some loss of precision. Then came ColBERT, which creates a contextual embedding for every word instead of one embedding per paragraph or page, and retrieves better than text and single-vector search combined, he said.
ColPali, whose name is a play on ColBERT, applies the same idea to images. A page is cut into small patches, and a visual language model trained on both images and text creates a contextual embedding for each patch. The same model embeds the user's text query into the same vector space, so the system can match a question directly to a region of a page.
Because nothing has to be extracted, the pipeline also gets simpler. Convert PDFs or Word documents to images, cut them into pieces, embed them "and you're done," Bratseth said. He was careful about how far it goes today: "I still say it's more promising than powerful."
What did the ColPali demo show?
In Vespa's public demo on a corpus of financial reports, a text query about trends in graduate applications retrieved the right page using only visual vectors, with no text indexing. A visualization showed where the model was looking. Clicking "trend" highlighted the graph, and clicking a digit in "2023" pointed to the year on the page.
A search for a child jumping in a puddle found an illustration, with attention on the child's outline and on the puddle. "It's actually looking at the images and understanding them," Bratseth said.
Why does text-only retrieval struggle with visual documents?
Extracting only text loses information in documents that mix text with pie charts and diagrams, Bratseth said. He pointed to finance, medical documents and construction, where most information sits in figures and drawings.
Ramchandani said RavenPack's research assistant, Bigdata.com, had handled charts the traditional way, with handcrafted rules running through OCR, extraction and vision language models. ColPali replaced a pipeline of six or seven steps that took about seven minutes, he said, with retrieval of the right piece of a rich document in about a second. In finance, he noted, only the people who built a chart usually know where to look in it.
Will ColPali replace OCR and parsing?
Not yet, according to Ramchandani. RavenPack plans to use this kind of system for retrieval, but reasoning still has to happen on top of what is retrieved. "I don't think OCR is going anywhere," he said, and a blend of both pipelines is what he described as the new way to build assistants.
Bratseth agreed that it "doesn't solve all problems." Models make human-like mistakes, such as confusing a handwritten one and seven, and there is no way to detect those errors at indexing time, he said. For dense visuals such as construction drawings, Vespa adds a short text description to each patch and is interested in combining several zoom levels in a single embedding. How many patches a page needs depends on its information density.
Asked about accuracy in medicine, Bratseth said visual models aren't mature enough for life-or-death applications. Ramchandani described RavenPack's stance as "trust but verify." The company presents retrieved information with source attribution, down to the exact piece of a PDF, and would not let an agent execute trades on its own.
How do you scale multimodal RAG?
ColPali produces a very large number of vectors. Bratseth said converting them to binary vectors, and shortening them, gives almost the same results while saving memory and CPU, since distances can be computed with Hamming distance rather than floating-point math.
Ramchandani's advice was to test first. Check whether vision models work in your domain using 10 or 20 PDFs, and see whether attention lands in the right place. Then work out the compute needed to generate embeddings, given the size of the archive and how often new documents arrive. For small collections, he said, a long-context model through an API may be enough.
Bratseth said serious deployments should start with evaluations built on your own data. "You don't really care how well it does on a benchmark, you care how well it does on your data for your problem," he said.
Looking ahead, both expect smarter visual language models. Ramchandani hopes retrieval models get smaller and expects that within about two years models will reason directly over audio, images, tables and text instead of converting everything to text first. His closing advice was to "first see if a simpler version of it works before you complicate your life," because costs "scale up in ways that will always shock you."
Keep learning in 2027
Data Science Connect has 24 free live webinars on its 2027 schedule, including AI Demo Days and Agentic AI Showcases where vendors show working systems. See the webinar schedule for upcoming sessions. Enterprise AI leaders also meet in person at The AI Enterprise Conference on Tuesday, August 31, 2027, at Pier Sixty in New York City.
Frequently asked questions
What is ColPali?
A retrieval approach that cuts document pages into image patches and uses a visual language model to create contextual embeddings for each, so text queries can match charts, tables and layouts directly.
How is ColPali different from ColBERT?
ColBERT creates contextual embeddings for every word of text; ColPali applies the same idea to patches of page images, Bratseth explained.
Does ColPali replace OCR in RAG pipelines?
Not according to RavenPack's Ramchandani, who said OCR and parsing remain necessary and that a blend of visual and text pipelines works best.
How do you scale ColPali to large document collections?
Bratseth recommended converting embeddings to binary vectors and shortening them, which saves memory and lets distances be computed with Hamming distance.
Watch: Transforming the Future of RAG with ColPali on YouTube
Event: Webinars · More coverage: Webinars insights
