Knowledge Graphs and LLMs: Grounding Answers with Neo4j
At COLLIDE 2023 in Atlanta, Neo4j's Phani Dathar said knowledge graphs can address LLM inaccuracy and missing company knowledge. He described a connect, enrich and consume approach that pairs vector search and generated graph queries with an LLM.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- LLMs need grounding. Dathar said LLMs give reasonable but not always factual answers and lack an organization's own domain knowledge.
- Graphs make relationships explicit. Knowledge graphs connect silos and add taxonomies and ontologies that give answers business context.
- Connect, enrich, consume. Build the graph, enrich it with derived relationships, algorithms and embeddings, then let the LLM consume it.
- Two retrieval patterns. Combine vector matches with graph relationships, or have the LLM generate Cypher queries from the graph schema.
- Completeness is the open problem. Database results are factual, but Dathar said coverage of complex questions still needs experimentation.
At COLLIDE Data Conference 2023 in Atlanta, Neo4j Director of Graph Data Science Phani Dathar argued that knowledge graphs can fix two of the biggest barriers to enterprise adoption of large language models: answers that sound reasonable but are not always accurate, and a lack of company-specific knowledge. His talk, "Contextual and Semantic Information Retrieval using LLMs and Knowledge Graphs," covered how to combine graphs with LLMs so answers are accurate, explainable and contextual.
Why do LLMs need a second data store?
Dathar said LLMs are "inherently probabilistic models" that give reasonable answers but not always factual ones, because they have only seen the data they were trained on and not an organization's own data. At the time, he said, enterprise LLM work was still mostly proofs of concept, with nothing concrete in production.
It was becoming common, he said, to pair an LLM with an additional data store that grounds responses in facts. Enterprise data is both unstructured and structured. Dathar said embedding text into vectors had made unstructured content much easier to search, but structured data, which he described as mostly facts, remained essential to businesses.
What does a knowledge graph add?
A graph connects data from different silos and makes the relationships between entities explicit, Dathar said. A knowledge graph goes further by adding taxonomies and ontologies that capture domain-specific meaning, which gives an LLM the context to respond in terms of the business.
He gave a recommendation example. With matrix factorization, he said, you boil purchases down to a user matrix and face the curse of dimensionality. In a graph it becomes a pattern match: start at the customer, probe the local neighborhood, find similar users and recommend products. Sometimes, he said, you don't need to go beyond one hop.
How does the connect, enrich, consume approach work?
Dathar framed the work in three phases. First, connect the data into a graph. Second, enrich it. Third, consume it, which is where the LLM comes in.
Enrichment can add derived relationships, such as linking two people who made similar purchases in subsequent store visits. It can also use graph algorithms such as community detection and clustering, or node embeddings that capture the structure around each node. In a cybersecurity example, he said, an analyst could probe the graph for nodes with high centrality scores to find data centers prone to cybercrime, rather than relying on generic LLM output.
How can vector search and graphs work together?
One common pattern, Dathar said, is to split PDFs into text chunks, store each chunk as a node with its vector as a property, and run semantic search with a native vector index. Neo4j includes vector indexing, so teams do not need both a separate vector database and Neo4j, he said in the question period.
Graph embeddings go a step further. Because they combine node properties with the topology around a node, he said, one vector carries "both contextual and semantic information." After finding similar documents, a team can traverse one, two or three hops to pull in connected facts.
What are the two retrieval patterns?
In the first pattern, a user's question goes to an embedding model, and the results are combined with explicit and derived relationships from the graph. An LLM then summarizes the results.
In the second, the LLM generates a Cypher query, Neo4j's graph query language, which runs directly against the database. Dathar described using LangChain's graph Cypher question-answering chain, with the graph schema and example queries attached to the prompt so the model does not invent schema elements. Results from the database are then sent back to the LLM, along with the question, for summarization.
Are the answers accurate and complete?
Asked how to verify accuracy, Dathar said that whatever comes back from the database is factual because the data in Neo4j is fact data. Completeness is the open question. He said his team was still experimenting with fine-tuning and with how many examples to give the model, noting that a patient-journey question about patients over 30 with diabetes and what happened 90 days after diagnosis requires a complicated query. This work, he said, depends on each customer's schema and questions.
He also said the team was exploring embedding graph neighborhoods, so a question can match a similar neighborhood rather than a single node.
His example was an insurance analyst asking about a hurricane. An LLM might report where it passed, but the useful answer is how many policyholders in the region face property loss.
Where to hear about data and AI next
COLLIDE sessions like this one are part of AI Connect's event program. COLLIDE 2027, the largest AI conference in the Southern U.S. with more than 1,000 enterprise data and AI leaders, takes place Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. See COLLIDE. AI Connect also runs 24 free live webinars on its 2027 schedule; see webinars.
Frequently asked questions
Why combine a knowledge graph with an LLM?
Dathar said LLMs give reasonable but not always accurate answers and don't know company data. A knowledge graph supplies facts, relationships and business context.
Do you need a separate vector database with Neo4j?
No, according to Dathar. Vector indexing is a feature of Neo4j, so vectors can be indexed there.
How does an LLM query a graph database?
The LLM generates a Cypher query using the graph schema and example queries in the prompt, and the database results go back to the LLM for summarization.
How accurate are the answers?
Dathar said data returned from the database is factual, but whether the answer is complete depends on the examples and coverage, which his team was still testing.
This article is based on remarks by Phani Dathar, Director Graph Data Science, Neo4j at COLLIDE Data + AI Conference 2023 on October 3, 2023. More coverage: COLLIDE AI Conference.
Filed under: Context, Semantics & RAG, Data & AI Infrastructure
