Vector Databases for AI: Cutting LLM Hallucinations with Zilliz
At COLLIDE 2023 in Atlanta, Zilliz developer advocate Yujian Tang explained why vector databases matter for AI applications. He covered retrieval augmented generation, semantic similarity, and why SQL databases and search libraries fall short at scale.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- RAG is the lead use case. Tang called retrieval augmented generation the most popular use case for vector databases.
- Ground the model in your data. A retrieval layer lets an LLM answer from an organization's own documents, which Tang said cuts hallucinations.
- Embeddings enable math on meaning. Vectors let search match differently worded sentences, images and audio.
- SQL and libraries fall short at scale. Tang said vector operations are too computationally intensive for traditional databases, and libraries lack production features.
- Benchmark on your own data. He pointed to VectorDBBench for comparing queries per second, cost and latency.
At COLLIDE Data Conference 2023 in Atlanta, Yujian Tang, a developer advocate at Zilliz, explained why AI applications need vector databases and how to evaluate one. Zilliz maintains the open-source Milvus vector database and the GPTCache project. Tang's talk covered unstructured data, retrieval augmented generation (RAG), semantic similarity and what separates a vector database from SQL databases and search libraries.
Why do AI applications need a vector database?
Tang opened with the claim that 80% of data is unstructured and 90% of that goes unanalyzed, a figure he called a trope. Vectors, he said, offer a way to use that data, whether it is text, audio or video. He described vector databases as critical to similarity search and "pretty much the only existing solution."
The most popular use case, he said, is retrieval augmented generation. Large language models hallucinate, Tang said, citing news about a lawyer who got in trouble using ChatGPT. He explained that GPT is a decoder-only architecture that predicts the next token from its inputs, so it produces plausible text rather than verified facts.
How do vector databases reduce hallucinations?
Tang described adding a data layer on top of the model. A user query goes to the vector database, which retrieves relevant information from the organization's own data, and the language model sends it back in a human-readable form. He said this lets teams build on their own data to cut hallucinations.
As an example, he pointed to OSS Chat, a Zilliz-built tool for chatting with open-source software. He also argued that querying large document stores is a major time sink for knowledge workers, so an LLM-based approach can be far more efficient.
What is semantic similarity?
Vectors are the internal representation a machine learning model produces for input data, Tang said. Documents, images or videos pass through an embeddings model, and the resulting vectors are stored in a database such as Zilliz or Milvus.
He used a toy example with the words queen, woman, man and king to show that subtracting one vector and adding another can land on a related word. The point was that "Vector embeddings allow you to do math on things that are not originally numbers," he said.
He also showed three differently worded questions about increasing profit. Keyword or fuzzy search would miss some of them, he said, while vector search returns all three. The same approach works for images, such as finding the closest match to a face in a database of celebrity photos.
Why not use a SQL database or a vector library?
SQL and NoSQL databases compare keys for exact matches, Tang said. Vector search looks for approximately similar data using long series of numbers, often hundreds, which means thousands or more calculations per query. Without purpose-built infrastructure, he said, a SQL database could be "spinning" for days. His summary: "Vector operations are too computationally intensive for traditional database infrastructures."
Vector search libraries such as FAISS and HNSW implementations are also not enough on their own, he said, because they lack the backup, multi-tenancy and reliability needed to scale, deploy and manage applications in production.
Tang, who works for Zilliz, said performance differences show up once a team reaches one million or ten million vectors. He described Milvus and Zilliz as designed for high-performance, billion-plus-scale use cases.
How can teams get started?
Tang pointed to the open-source VectorDBBench tool, which lets teams benchmark on their own data and produces graphs for queries per second, queries per dollar, throughput and latency. He also listed Zilliz's vector database 101 series and RAG tutorials, the open-source Milvus repository, and Zilliz Cloud, a fully managed service with a free tier at the time.
What did the audience ask?
Asked what a photo embedding looks like, Tang said it depends on the model. He cited OpenAI embeddings at 1,536 dimensions, MPNet at 768, and MiniLM models at 384. Embeddings can be compared only if they have the same dimension, he said, and a single multimodal model gives more consistent results than comparing across two models.
Asked about probabilities behind the vectors, he said embeddings are derived from a machine learning model and depend on the model and data supplied.
Where to hear about AI data infrastructure next
COLLIDE 2027, the largest AI conference in the Southern U.S. with more than 1,000 enterprise data and AI leaders, takes place Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. See COLLIDE. AI Connect also hosts 24 free live webinars on its 2027 schedule at webinars.
Frequently asked questions
Why do LLMs need vector databases?
Tang said LLMs hallucinate, and a vector database lets them retrieve relevant information from an organization's own data before answering.
Why not use a SQL database for vector search?
Tang said SQL databases do exact key matching, while vector search needs many calculations over long number series, which traditional infrastructure handles poorly.
Why not just use a vector search library?
He said libraries lack backup, multi-tenancy and reliability needed to scale and manage applications in production.
How many dimensions do embeddings have?
It depends on the model, Tang said: for example OpenAI embeddings have 1,536, MPNet 768 and MiniLM 384.
This article is based on remarks by Yujian Tang, Developer Advocate, Zilliz at COLLIDE Data + AI Conference 2023 on October 3, 2023. More coverage: COLLIDE AI Conference.
Filed under: Data & AI Infrastructure, Context, Semantics & RAG
