RAG that works in a pilot can turn slow, costly and leaky at enterprise scale. In a Data Science Connect webinar, experts from Tecton, Nexla and Novartis explained how to cut RAG costs, keep indexes in sync, enforce chunk-level access control and evaluate accuracy honestly.
Key takeaways
- Define your kind of scale. A few large RAG systems and many small ones need different solutions, Wang said.
- Match serving to workload. Batch jobs served through online infrastructure can cost far more than expected.
- Use smaller models where you can. Desai said routing most steps to cheaper models cuts cost and latency.
- Sync incrementally and watch index quality. Deletions are hard and repeated updates degrade indexes, so plan reindexing.
- Enforce access at the chunk level. Tie existing access controls to documents and chunks, and scan for confidential content.
Retrieval-augmented generation (RAG) that works in a proof of concept can get slow, expensive and leaky once it reaches a whole enterprise. In the Data Science Connect expert panel webinar "The Future of Retrieval-Augmented AI: A Blueprint for Scalable and Cost-Effective RAG Systems," Han Wang, lead engineer for generative AI at Tecton; Amey Desai, head of AI at Nexla; and Naveen Pentakota, associate director of data science at Novartis, discussed how to scale RAG without losing control of cost or security. The webinar was sponsored by Tecton, Galileo and Nexla, and Brian Mink, co-founder and president of Data Science Connect, moderated.
What is RAG and why does it matter for enterprises?
Desai compared large language models to what SQL did for tabular data: a tool for asking questions of unstructured data such as PDFs, audio and images. Pentakota described public LLM training data as an ocean with enterprises as islands within it, each using LLM capabilities to build on its own data.
Wang said RAG gives a model the unseen, use-case-specific context it needs on top of its base knowledge. He cautioned that RAG is often equated with semantic or similarity search, which is only one retrieval method among several that are "equally important."
What makes RAG hard to scale?
Wang said the first step is defining scale. A small number of large RAG systems and a large number of small ones need different solutions, and he said he had never seen a case with many large ones. For a large RAG, he suggested metadata filtering, partitioning or simply breaking it into smaller ones.
Pentakota pointed to growing data volume, variety and velocity across both the retrieval and generation pipelines, while keeping systems responsive and data private. Desai added two more: querying structured, semi-structured and unstructured data together, and collaboration between the engineers who build a RAG system and the business users who rely on it. Even with LLM-as-a-judge methods, he said, "human evaluation is probably the best thing that we can still do right now."
How can organizations reduce RAG costs?
The panel offered a range of tactics:
- Match the serving pattern to the workload. Wang said running 10 million queries against a large index is a batch job, and serving it through online infrastructure can cost far more than expected. Static data can be indexed more slowly in exchange for faster serving.
- Route tasks to smaller models. Desai said models have become an order of magnitude cheaper, and many steps can use a smaller model, saving the largest one for critical pieces.
- Compare open-source deployments. Desai suggested benchmarking an open-source model on serverless functions against hosted models, accounting for bursty traffic.
- Specialize and compress. Pentakota cited domain-specific models for fields like healthcare and finance, plus pruning, quantization and distillation.
- Count the people. Desai noted that self-hosted models carry development, deployment and maintenance costs.
How do you keep a RAG index in sync with enterprise data?
Connectors to enterprise systems come first, Desai said, followed by incremental computation. Updating a folder of documents shouldn't trigger re-embedding the whole collection. He also expects migrating indexes between vector databases to become a real concern.
Wang said continuous sync pipelines are stateful and hard to maintain, and deletions are harder than additions. Index quality also degrades with each incremental update, so teams have to monitor quality and decide when to rebuild the index. On who curates the data, Desai said it takes data engineers working with domain experts, with automated pipelines rather than manual uploads from a laptop.
How do you enforce access control in RAG?
Wang recalled a case from his previous role leading Lyft's AI and ML platform: 10,000 documents, with each user allowed to see a different subset. Replicating indexes per role is simple but hard to maintain, he said. Metadata filtering works, but heavy filtering can reduce nearest-neighbor search quality unless the vendor handles it specially.
Desai described three parts: tying existing access controls from systems such as Microsoft or Google Workspace to documents, enforcing them at the chunk level, and sending user group information with each query. "Chunk level ACLs I think is a bare minimum," he said. He also recommended scanning for confidentiality, since internal presentations on strategy or revenue can end up answerable by anyone with a chatbot. Pentakota stressed role-based access, regulations such as GDPR, CCPA and HIPAA, and real-time checks for prompt injection.
Where is RAG heading, and is it accurate enough?
Wang said unstructured data's importance is overstated and structured data's is understated, and expects better ETL tooling. He sees RAG increasingly used as a tool inside agentic systems, as his teams have done. Desai expects more prescriptive, task-specific assistants and faster retrieval. Pentakota predicted multimodal and multilingual RAG.
On latency, Wang said generation, not retrieval, usually dominates, so domain-specific fine-tuned models help most. Desai recommended dynamic chunking and smarter metadata filtering. Pentakota said streaming responses reduce perceived latency.
On accuracy, Wang said "nothing is 100%," and the real need is unbiased evaluation that covers what the system should and shouldn't answer. Desai called for page-level citations and setting user expectations. Pentakota said "the question you ask is as important as the answer you get."
In closing, Wang said "there is no single solution that can solve all scenarios," and Desai urged teams to "calculate the cost per question."
More on enterprise AI architecture in 2027
Data Science Connect has 24 free live webinars on its 2027 schedule across the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. See the webinar schedule. Technical leaders can also meet in person at The AI Enterprise Conference on Tuesday, August 31, 2027, at Pier Sixty in New York City.
Frequently asked questions
How can you reduce the cost of a RAG system?
Panelists recommended matching batch or real-time serving to the workload, routing tasks to smaller models, considering open-source deployments and tracking cost per question.
How do you handle permissions in an enterprise RAG system?
Desai said to inherit existing access controls, enforce them at the chunk level and pass user group information with each query; Wang noted heavy metadata filtering can reduce search quality.
Why is keeping a vector index in sync difficult?
Wang said continuous sync pipelines are stateful, deletions are harder than additions, and repeated incremental updates degrade index quality until the index is rebuilt.
Is RAG accurate enough for business use?
Wang said nothing is 100% accurate and teams need unbiased evaluation; Desai recommended page-level citations and setting user expectations.
Watch: The Future of Retrieval-Augmented AI: A Blueprint for Scalable and Cost-Effective RAG Systems on YouTube
Event: Webinars · More coverage: Webinars insights
