Searching video and images means keeping context that flat vectors lose. In a Data Science Connect webinar, Vespa.ai's Bonnie Chase and Zohar Nissare-Houssen explained tensors, co-located compute and multi-phase ranking, then demoed a hybrid query that found a single scene inside a cartoon.
Key takeaways
- Vectors alone lose context. Image vectors drop spatial relationships and video vectors collapse time, Chase said.
- Tensors keep structure. Multi-dimensional tensors store per-chunk, per-token or per-patch embeddings plus metadata like timestamps.
- Move compute to the data. Co-locating data, indexes and models avoids slow network transfers at scale.
- Rank in phases. Cheap lexical or quantized retrieval first, then fine-grained embedding comparison on fewer candidates.
- Context narrows the search. Hybrid queries combine metadata like genre or camera feed with scene-level semantic search.
Enterprises are moving past text-only retrieval, and the hard part is keeping context when the data is images or video, according to two Vespa.ai leaders. In "Leveraging Data Beyond Text: Multi-Modal AI at Scale," a Data Science Connect webinar sponsored by Vespa.ai and hosted by DSC co-founder and president Brian Mink, Bonnie Chase, Director of Product Marketing, and Zohar Nissare-Houssen, Strategic Presales Lead, explained how tensors, hybrid queries and multi-phase ranking support search across video at scale. Nissare-Houssen closed with a live demo that found a specific scene inside a cartoon.
Why is multimodal retrieval harder than text search?
Chase said multimodal systems need three things: scale to index billions of documents, images and videos; accuracy across every modality; and context. Context is where multimodal RAG usually struggles, she said, because "vectors alone are not enough."
She described the search market as two camps. Text-first engines added vector search, and newer vector databases added hybrid search. Vespa, she said, started with hybrid and vector search in 2014.
She illustrated scale with customer examples. Spotify uses Vespa to index every word of its podcasts, so listeners can find an episode by describing what was said. Perplexity uses it for retrieval behind its answer engine; Chase said it has indexed more than 1.5 billion documents and aims for 10 billion. Vinted, a secondhand marketplace with more than a billion items, uses it for personalized search and recommendations, and a child-safety app uses it to flag harmful text, images and memes within milliseconds.
How do you run AI search with low latency at scale?
Most systems store data in one place and process it in another, Chase said, and moving large volumes across the network breaks down at scale. Vespa inverts that. "We bring the computation to the data," she said. Data, indexes and machine learning models sit on the same nodes, so ranking, filtering and embedding comparisons happen locally.
Every write is real time, she said, with partial updates to fields such as prices, inventory or behavior signals, and no index rebuilds. Ranking runs in phases: lightweight scoring first, heavier models for the most promising candidates, with multiple rank profiles over the same data for A/B testing or separate search and recommendation logic.
What are tensors, and why do they matter for multimodal AI?
A vector is a flat, one-dimensional array of numbers. A tensor is multi-dimensional, which Chase called the next stage beyond vectors. Flattening other modalities into a single vector loses information, she said. "An image vector, for example, will lose spatial relationships," and "a video vector collapses time," so frames can't be separated. Text vectors lose token-level detail.
Tensors can hold token- or patch-level embeddings, separate dimensions per modality such as frame or region, and metadata like timestamps and bounding boxes. Late-interaction methods such as ColBERT for text and ColPali for document images need multiple embeddings per document, which tensors can store natively, she said.
How does video retrieval work in Vespa?
Nissare-Houssen said he is seeing enterprise demand for video retrieval. One global media company wants buyers to search inside the content it resells, not just its keywords and metadata. An advertising analytics company wants to measure exactly when, and for how long, a sponsor's logo appeared during a live broadcast.
His demo used models from Twelve Labs. A video understanding model generated a five-sentence summary and keywords for each video. A multimodal embedding model split each video into six-second chunks and produced a 1,024-dimension embedding per chunk, capturing scene, audio and on-screen text. Each video became one Vespa document holding metadata and a tensor of chunk embeddings.
That design enables what he called contextualized retrieval. A plain embedding search for a Grand Canyon sunset returns hundreds of documentary clips. A hybrid query can add context such as genre, year or actors, then find the exact segment. For surveillance video, he said, that context could be a camera feed, a time range or a location.
What did the demo show?
Working from a public Jupyter notebook with three cartoons from the Internet Archive, Nissare-Houssen asked for Santa Claus on his sleigh. The first ranking phase ran a lexical search over titles, keywords and summaries. The second compared the query embedding with every six-second chunk. "Ranking is computationally expensive," he said, so the two-phase approach keeps accuracy high without sacrificing performance.
The query returned "'Twas the Night Before Christmas" with the best-matching segments and their timestamps. Playing from the top result brought up the sleigh and eight tiny reindeer.
Asked how to balance latency and cost, he suggested using lower-dimension or binary-quantized embeddings for coarse first-phase retrieval, including Matryoshka-style dimension reduction where the model supports it, then reranking a smaller set at full precision.
Keep learning on DSC webinars
Data Science Connect has 24 free live webinars on its 2027 schedule across four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. See the full lineup on the webinars page.
Frequently asked questions
What is the difference between a vector and a tensor in AI search?
A vector is a flat, one-dimensional array of numbers; a tensor is multi-dimensional, so it can hold many embeddings per document, such as one per six-second video chunk, along with structure like time or position.
How does multimodal video retrieval work?
In Vespa's demo, a video model generated summaries, keywords and an embedding for each six-second chunk; a hybrid query then ran lexical search on the metadata and semantic search across chunks to find the exact segment.
What is contextualized retrieval?
Combining structured context such as genre, year, actors, camera feed or time range with semantic scene search, so results match the user's intent instead of every visually similar clip.
How do you balance latency and cost in multimodal search?
Zohar Nissare-Houssen suggested lower-dimension or binary-quantized embeddings for first-phase retrieval, then reranking a smaller candidate set at full precision.
Watch: Leveraging Data Beyond Text: Multi-Modal AI at Scale on YouTube
Event: Webinars · More coverage: Webinars insights
