AI Governance: Layered Defenses and Guardrails, Per DataRobot

COLLIDE Data + AI Conference 2024Governance in the Era of AIWatch the full session · 20 min

At COLLIDE 2024 in Atlanta, Jessica Lin, Lead Data Scientist at DataRobot, argued that enterprises should treat generative AI governance as layers of imperfect defenses rather than a single fix. In a session titled "Governance in the Era of AI," sponsored by DataRobot, she walked through regulation, tooling sprawl, compliance documentation and runtime safeguards. COLLIDE ran October 10-11, 2024, at the Sandy Springs Performing Arts Center.

Lin described DataRobot's approach as she presented it, so product claims below are attributed to her.

Why are enterprises nervous about releasing generative AI?

Lin said her conversations with teams in the field kept returning to governance. Leadership wants generative AI, she said, but teams are nervous even about releasing something internally, "the lowest level of risk," because the perceived risk is still high.

Regulation adds to the difficulty. She pointed to three influential regulatory frameworks across global regions, noting similarities and differences among them, plus state-level rules in the United States, citing California, Colorado and Utah as examples. DataRobot has a team focused solely on regulations and ethics, she said, but for most organizations that is not anyone's day job.

What is the Swiss cheese model for AI defense?

Even with perfect knowledge of the rules, Lin said, implementation is hard. Mainstream platforms, in her view, require teams to piece together many tools and microservices, making it hard to see the whole architecture or find weak points.

Her analogy was Swiss cheese. "There's never going to be one perfect defense solution," she said. Each layer has a focus and caveats, but stacking them gives a better defense. She organized the layers around three components: people, processes and auditability.

• People: Data science builders, IT, compliance and model risk teams all touch AI, so the platform has to be accessible to each of them.
• Processes: Each new workflow or tool adds risk that is harder to oversee, so teams should be wary of an endless sprawl of bespoke solutions.
• Auditability: Organizations need to be able to ask what happened, who pushed changes and who is affected, and to have approval policies that are repeatable as AI efforts grow.

How should companies avoid lock-in and keep humans in control?

Lin laid out three examples of DataRobot's focus, which she said is on governance rather than building foundation models. The first was avoiding lock-in to a single ecosystem. The field changes so quickly, she said, that organizations should want something model agnostic, where predictive and generative models, built in DataRobot or elsewhere, are registered in one governance hub.

From there, teams can set approval policies so that, as she put it, "nothing ever gets pushed to production without having human eyes on it."

What goes into AI compliance documentation?

The second example was compliance documentation. Lin said DataRobot has worked with financial services teams that have active model risk management groups, and that AI regulation may bring similar processes to more companies.

She said the platform can generate documentation covering the use case, inputs, assumptions and how the chosen approach compares to alternative "challenger" approaches, including the caveats of a simpler model.

For generative AI, she added the concept of attack vectors. Teams can test how well an application stands up to attacks such as phishing or prompt injection, get an objective score, and include it in documentation. That lets teams be "very explicit and objective" about what they are putting out, she said.

How do guardrails and moderation protect generative AI apps?

The third example was runtime safeguards. Lin described cost risks, such as a denial-of-service attack or an automation system gone rogue that could cost hundreds of thousands of dollars before anyone notices, and quality risks from responses no human has reviewed.

She distinguished monitoring from moderation. Monitoring tracks cost, responses and user satisfaction over time and helps reveal use cases that consistently produce bad results. Moderation acts in the moment. Guards can check responses for consistency, keep a bot on topic, block output from reaching the user, or detect personal information such as a phone number and stop it from reaching the language model at all. Misinformation in a response can be redacted or replaced with a request for clarification.

Lin summed up the goal as layering governance and compliance across all levels of models and observing AI assets in production, whether teams build in DataRobot or elsewhere.

Where can leaders keep exploring AI governance?

COLLIDE 2027, the largest AI conference in the Southern U.S., will bring 1,000+ enterprise data and AI leaders to the Sandy Springs Performing Arts Center in Atlanta on Wednesday, October 13, 2027. Details are on the COLLIDE event page, and registration is coming soon.

Frequently asked questions

What is the Swiss cheese model for AI governance?

Lin used it to say no single defense is perfect; layering several imperfect controls gives a stronger overall defense.

What should AI compliance documentation include?

Lin said it can cover the use case, inputs, assumptions and comparison with challenger approaches, plus attack-resistance test scores for generative AI.

What is the difference between AI monitoring and moderation?

Monitoring tracks cost, response quality and user satisfaction over time; moderation blocks or redacts problem inputs and outputs in real time.

Why avoid locking into one AI ecosystem?

Lin said the field changes quickly and lock-in limits control over pricing and customization.

This article is based on remarks by Jessica Lin, Lead Data Scientist, DataRobot at COLLIDE Data + AI Conference 2024 on October 10, 2024. More coverage: COLLIDE AI Conference.

Filed under: AI Governance & Trust, Security & Fraud