Leaders from SAP, Skyflow and Fivetran, with a General Mills moderator, laid out the architecture behind AI that scales: isolation boundaries, model-agnostic interfaces, agents that survive tool failures, and observability that tracks why agents touch data.
Key takeaways
- Build focused agents, not super agents. A narrow scope with a human in the loop at the right point resolves tasks quickly.
- Isolate teams and decouple models. One team's misconfiguration shouldn't affect others, and swapping an LLM shouldn't mean rewriting code.
- Design for failure. Agents that get error feedback can route around a failed tool instead of failing the whole loop.
- Unify batch and real time. Trivedi recommends one pipeline logic with two execution modes, plus semantic caching for repeat questions.
- Observe why agents access data. Log what was accessed, by whom and why, and feed that back into governance automatically.
Plenty of companies can build one AI proof of concept. Running 50 of them at once without things breaking is the harder problem, SAP's Shubham Trivedi said in the Data Science Connect webinar "Scaling AI: Designing Systems Built to Grow," sponsored by Skyflow, SAP and Fivetran.
Trivedi, Digital Solution Advisor at SAP, joined Joe McCarron, Head of Developer Advocacy at Skyflow, and Kelly Kohlheffer, Senior Director, Global Partner Engineering at Fivetran. Sanchit Srivastava, a senior manager for architecture and analytics at General Mills, moderated.
What architecture do you need to scale AI from proof of concept to production?
Each panelist named the components they consider essential. Kohlheffer started with agent-native data access and a human in the loop at the right point. He described a support agent that checked his smart ring's telemetry, confirmed his shipping details and closed a replacement case in under three minutes, with Kohlheffer himself as the only human in the loop.
The lesson he drew was scope. "Don't build super agents, build focused agents," he said, describing his philosophy for the past two years.
Trivedi listed three non-negotiables:
- Isolation boundaries, so that "one team's mess becomes everyone's problem" doesn't happen when a misconfiguration or budget overrun hits a shared environment
- A model-agnostic API interface, so that swapping an LLM never requires rewriting the code above it
- Evaluation before go-live, including expected users, token usage and budget, because AI failures happen silently as models keep responding, but less accurately
McCarron started from reliability and predictability. That means observability and graceful degradation when a vendor's API goes down. "You don't want their SLAs to be your SLA," he said of model providers.
How should AI systems handle tool failures and long-running agents?
McCarron said LLMs make a new kind of resilience possible. If an agent gets error information back when a tool fails, it can steer itself to other tools. "You shouldn't have to fail the loop if the single tool fails," he said.
He also flagged variable-length loops. Teams that once considered a one-second HTTP request long now have requests streaming data back for 15 minutes or more. Many are building one real-time system and one background system, he said, when a single system that supports both will pay off.
Kohlheffer said he saw this work. When an agent's semantic view couldn't answer his question, an MCP server connected to the source's API let the agent route around the gap and return an answer about 20 seconds later.
Why does modularity matter for enterprise AI?
Trivedi said modularity has to go below the team level into the code. A grounding module produces an output that the prompt template consumes, and the template shouldn't care whether that grounding came from SharePoint, SAP Datasphere or a vector table. He compared the result to LEGO blocks that can be removed and replaced without disturbing those above or below.
McCarron, who spent most of his career as a product leader for APIs, said contracts and versioning are old ideas that agents have made critical. He urged teams to own their abstractions rather than take whatever interface a vendor provides, and to evaluate how output changes when one model replaces another.
On the data side, Kohlheffer argued for landing raw data once in open formats such as Iceberg, Delta and Parquet, so any compute engine or agent framework can query the same data set.
Should real-time and batch AI pipelines be separate?
Trivedi said no. Teams should "stop treating real-time and batch orchestration as two different architectures," he said, and instead run the same pipeline logic in two execution modes. He pointed to semantic caching, which retrieves stored answers by meaning, and said about 70% of enterprise queries are variations of questions already asked, such as routine HR questions about leave.
McCarron agreed and framed it as user experience. Users want one interface that can either start a long background task or route a quick question to a small model, he said, without exposing the system's internal boundaries.
What should you observe in a production AI application?
McCarron said the top priority is getting evaluations right early, with automated evals that define what good looks like. For security and privacy, teams need to log what data is accessed, by whom and why. Agents can generate a structured justification for each access, he said, which matters under privacy rules such as GDPR, and that log should feed an automated loop that locks down improper access without waiting for a human review.
Kohlheffer added tracking of the tools an agent called and the confidence it expressed at each step, warning that agents can be very confident when they're wrong. Trivedi listed three layers: infrastructure uptime, model quality scores and RAG groundedness, meaning whether the model chose the right documents.
Common mistakes when scaling AI
Asked about the most common scaling mistake, Trivedi said it is treating governance as an afterthought. McCarron offered a "hot take": "Too many teams optimize into smaller agents before they need it." Factor out sub-agents when a main agent's quality drops, he said, not before.
In a closing lightning round, McCarron chose "healthy skepticism" toward trends. Trivedi said "governance is your architecture." Kohlheffer chose a deterministic core with a controlled probabilistic perimeter, built on "a data foundation engineered to last."
More on enterprise AI architecture
Data Science Connect has 24 free live webinars on its 2027 schedule across four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. See the webinar schedule. Practitioners can also meet in person at The AI Enterprise Conference on Tuesday, August 31, 2027, at Pier Sixty in New York City.
Frequently asked questions
What architecture is needed to scale AI?
The panel named isolation boundaries between teams, model-agnostic interfaces, evaluation before go-live, observability and graceful degradation when a vendor API fails.
Should AI agents be small and specialized?
Kohlheffer advises focused agents over super agents, while McCarron warns against splitting into sub-agents before a main agent's quality actually drops.
What is semantic caching in AI?
Caching that stores and retrieves answers by meaning and context rather than exact keywords, so repeated variations of a question skip an LLM call, as Trivedi described it.
What should you monitor in a production AI application?
Infrastructure uptime, model output quality through automated evals, RAG groundedness, the tools an agent called and what data it accessed and why.
Watch: Scaling AI: Designing Systems Built to Grow with SAP, Skyflow & Fivetran on YouTube
Event: Webinars · More coverage: Webinars insights
