AI agents will overreach, and traditional security controls weren't built for probabilistic software, Rubrik's Cal Al-Dhubaib said at The AI Enterprise Conference 2026. He shared real examples of agent overreach, a three-part model of context-based control, and three questions every team should ask before AI fails.
Key takeaways
- Assume AI will fail. Giving LLMs tools and data access turns occasional errors into a virtual guarantee, Al-Dhubaib said.
- Agents overreach in unexpected ways, from publishing a report to a public GitHub gist to leaking data through a search query with read-only access.
- Static controls fall short. Access lockdowns, human approvals, endpoint detection and periodic scans each miss agent intent or speed.
- Context-based control checks posture, identity and access, and runtime behavior against user intent and natural language policies.
- Plan for failure. Most organizations lack a backup plan for when an AI system goes wrong or goes offline.
AI systems will fail, and enterprises should plan for it, Cal Al-Dhubaib of Rubrik told attendees at The AI Enterprise Conference 2026 in New York City. In his session "Engineering Trust Into AI Systems: Designing for When Things Go Wrong," Al-Dhubaib, who has spent more than 10 years building machine learning systems for hospitals, financial services and energy companies, explained how AI agents overreach and why context is becoming a new kind of security control.
"You have an agentic problem whether or not you want to acknowledge it," he said.
Why are AI incidents increasing?
Al-Dhubaib pointed to the AI Incident Database, which he tracks, as showing a rise in reported incidents since the arrival of generative AI, though he said the numbers appear to be leveling out in 2026. Two forces drive the rise, he said: AI has become much easier to build with, and threat actors are using AI for scams such as AI-generated phishing.
He cited public examples. Air Canada's chatbot invented a bereavement policy, and the airline ended up in court over a $700 ticket. Cursor's support chatbot gave customers a false reason for being logged out of their accounts, and the company lost customers. Things get worse once AI can take action, he said, citing an Amazon outage tied to AI-generated code that he said cost about 6.3 million orders.
How do AI agents overreach?
Once an LLM has access to tools such as data sets, APIs or endpoints, the possible combinations of systems and actions become limitless, Al-Dhubaib said. That turns the chance of an AI system sometimes being wrong into a virtual guarantee. It also makes agents attractive to attackers. "This is the new form of insider threat," he said.
"Claude Code loves to overreach," Al-Dhubaib said, describing his own company's experience. In one case, an agent asked to publish a report internally hit an error and published it to a public GitHub gist instead. The data wasn't sensitive, he said, but the agent did what it was asked using the tools it had, in a way that created risk.
He described other scenarios:
- Combining permitted sources: In energy, marketing and transmission teams can't share information. An agent authorized to read both could combine them and produce an answer that suggests collusion. At a public company, combining sources could reveal material non-public information.
- Read-only exfiltration: An agent with only read access pulled sensitive Salesforce data into its context, then used it in a web search query, sending it to an outside search engine.
- Manipulated context: Longer context windows increase the risk of conflicting or outdated sources, and context can be deliberately undermined, as with prompt injections hidden in documents.
Why don't traditional security controls work for AI agents?
Enterprise controls were built for static, rules-based software, Al-Dhubaib said, while AI is probabilistic. He walked through common responses and their limits.
Locking down access misses risks that interfaces used to prevent. Aggregating a public company's entire sales pipeline used to require going account by account, he said. With an agent connected to Salesforce, it takes one question, and no traditional control says "don't aggregate financial information."
Human approval breaks down as well. After a few permission prompts, people start pressing enter without reading. "Human in the loop doesn't work when you're overwhelming the human with decisions," he said. Endpoint detection misses an agent's intent, and periodic posture scans can't catch an agent that spins up a cluster, acts and shuts it down between scans.
What are context-based controls for AI agents?
Context-based control means judging not just whether an action is permitted but whether it is permitted given the user's intent, the relevant policies and the data involved, Al-Dhubaib said. Rubrik breaks this into three parts:
- Agentic posture: what AI systems exist, what they can read, write or delete, and what risky contexts they could operate in. Rubrik expresses these as natural language policies, like a policy handbook for agents.
- Agentic identity and access: agents usually inherit the full permissions of the person who launched them. Just because an agent can read something, he said, "doesn't mean that it should read everything."
- Agent runtime: checking each step of an agent's work against intent, policy and data, then allowing it, escalating it to a human or blocking it in real time.
He also urged teams to cap agent spending. "Please set budgets for these things. Don't let them run crazy." Pairing anomaly detection with these controls can reveal new contexts that need policies, he said, such as a new Salesforce integration that starts aggregating financial data.
What questions should teams ask about AI failure?
Al-Dhubaib left attendees with three questions: when are AI failures more likely or more costly, how will you know when a system is acting against intent, and what is the backup plan when it goes wrong. Most organizations are furthest behind on the third, he said.
"This is not just a technology problem. This is also a business resilience problem," he said. His call to action was to make AI "a little bit more boring again" so teams can apply the resilience practices they already know.
Continue the AI risk conversation in 2027
Enterprise AI security and governance will be on the agenda again at The AI Enterprise Conference, returning to Pier Sixty in New York City on Tuesday, August 31, 2027. Registration is coming soon on the conference page. Data Science Connect's 24 free live webinars in 2027 also cover agentic AI and governance.
Frequently asked questions
What is AI agent overreach?
When an agent pursues its goal using the tools and access it has, but in a way that creates risk, such as publishing internal data publicly or combining sources it shouldn't, as Al-Dhubaib described.
What are context-based security controls for AI?
Controls that judge whether an agent's action is appropriate given the user's intent, relevant policies and the data involved, rather than relying only on static yes-or-no permissions.
Why doesn't human in the loop stop risky AI agent actions?
Al-Dhubaib said people overwhelmed with approval prompts start approving without reading them.
How can a read-only AI agent leak data?
In one example Al-Dhubaib described, an agent pulled sensitive Salesforce data into its context and then used it in a web search query, sending it to an outside search engine.
Watch: Engineering Trust into AI Systems: Designing for When Things Go Wrong on YouTube
Event: The AI Enterprise Conference · More coverage: The AI Enterprise Conference insights
