Runtime governance for agentic AI: A practical guide

The gap between “we have a policy” and “the policy runs in evaluated code” is where agentic AI breaks production. University of Edinburgh research identifies a specific failure mode: agentic AI harms emerge from execution trajectories (unsafe tool use, skipped approvals, uncontrolled side effects) that pre-deployment model checks cannot catch.

This guide walks through the runtime controls that close that gap, in the sequence you’d build them.

TL;DR

• Pre-deployment checks miss agentic failures; governance runs at execution time.
• Flag conditions encode rollout policies so every agent change inherits them.
• Approval gates need upfront criteria; reactive expansion stalls agent velocity.
• Gate rollout exposure on production signals, not developer confidence.
• Stale flags accumulate unexpected interactions; cleanup is a governance requirement.

Encode your rollout policies once, so every agent change inherits them

The first control is the base for all subsequent layers: policy-as-code. An agent that generates and deploys a feature has no reason to consult a written runbook. If your governance policies exist only as documentation, they are bypassed by default every time an agent ships.

Activation strategies like gradual rollouts and targeting rules become the governance mechanism when you encode policies at the flag evaluation layer. Every agent-generated flag inherits these policies automatically. The agent creates the flag; the platform applies the policy. No separate review step required.

This logic drives FeatureOps: managing software behavior at runtime regardless of who wrote the code. Whether a feature came from a senior engineer or an AI agent, it passes through the same flag evaluation gate.

Pair this with trunk-based development. Merge AI-generated code to main frequently and keep incomplete features hidden behind flags. That maintains a clean production surface while agents iterate.

Require approvals on high-risk changes with change requests

Some agent actions require a human signature before they take effect. Define “high-risk” before you need it. Database migrations and permission escalations require a pause-and-resume gate. Changes touching more than one service boundary need more than a simple flag evaluation.

The principle is continuous authorization: check the agent’s authority at each decision boundary, not just at session start, an approach formalized in research like the MI9 protocol. Pair it with a fail-closed default: if a governance check fails or visibility is interrupted, the agent’s action capability is revoked rather than granted.

In practice, the agent reaches the high-risk action class and stops. The Unleash runtime control architecture maps this as a pause-and-resume pattern: execution holds until a human approves.

Scope what agents can touch with role-based access

An agent with broad standing permissions can still cause real damage even after receiving approval for one action. Task-scoped access limits surface area by binding tool access to the assigned task context rather than a persistent capability set.

Safous’s research indicates that access must be tied to the agent’s role, assigned task, and operational context. Standing access creates a risk that single-action review misses: a sequence of individually low-risk calls can assemble a privileged workflow. Reading config files, identifying system state, then requesting access to a management interface: no single call would have triggered an alert. The chain is the problem.

The fix is to check the sequence, not just the call: verify that the agent follows the expected step order, so tool-chaining risks that individual action checks would miss get caught. In flag terms, you express these scopes as targeting rules tied to the agent’s identity. A flag enabled for “database-migration-agent in staging” is not automatically enabled for “feature-build-agent in production.” The scopes are defined separately and evaluated at runtime.

Roll out on production signals, not vibes: safeguards and impact metrics

Agents can merge changes faster than any human team could review them. The gating condition for exposure should be production signals, not calendar time or developer confidence.

What to instrument

Focus on metrics that degrade when something is wrong. Error rates and latency budgets are the floor. For AI-specific output, add token spend caps to catch runaway loops and drift signals that flag when an agent’s behavior diverges from its intended goal. When that signal fires, the rollout pauses.

What a pause looks like

A pause functions as a flag evaluation that stops returning true for the next exposure cohort until the degraded signal recovers. The change is still merged. The code is still in production. The exposure percentage stops advancing, or even goes to zero.

Tink, a Visa solution, demonstrated this at scale. The team used Unleash to toggle features off instantly across 25-plus services and 20 environments, limiting the scope of impact when problems surfaced in one part of the system. The Tink case study shows the result: surgical response, not a full-system revert.

Default thresholds

Start conservative. Set a 1 percent initial cohort advancing on 24-hour signal windows. That approach gives agents room to ship frequently while maintaining safety. You only push to 100 percent after observing stable production signals. Tighten or loosen those windows as your team builds confidence in the signals you are tracking.

Keep a complete audit trail of every agent action

HTTP logs record that a call happened. Semantic telemetry records why the agent made it: the intent and the governance state behind each action. For regulated industries, that’s the compliance record. For engineering teams, it’s the difference between knowing what broke and knowing why.

Semantic telemetry captures the agent’s reasoning alongside the action output. Every flag evaluation, every approval event, and every rollback must be timestamped and tied to the agent identity that triggered it. Not just “flag X was toggled at 14:32.” Flag X was toggled at 14:32 by agent Y, following approval event Z, after rollout safeguard on metric M triggered a pause.

For example, Prudential built a custom ServiceNow integration so developers toggle features in production without opening a manual ticket, and all changes and approvals sync automatically in the background to satisfy audit requirements. Auditors see a complete record. Developers see nothing additional in their workflow.

Clean up stale flags before AI output buries you

Agents generate flags faster than humans retire them. That backlog creates a class of problems the prior five controls don’t address.

Two unrelated flags can interact at evaluation time. When multiple agents modify overlapping parts of the codebase, the combinations multiply: a flag for a database connection pool setting may evaluate alongside a new flag that changes connection handling for an AI-generated feature. Neither flag’s owner anticipated the overlap, and unexpected flag interactions are not theoretical. At scale, they’re a predictable incident class.

Lifecycle management is one of the four FeatureOps pillars for this reason. The practical rule: retire a flag when its feature has shipped to 100 percent, the kill-switch case is closed, and no other flag references it. Unleash tracks the lifecycle state for every flag. And Unleash’s MCP server can surface the stale flag list to an AI coding agent, making cleanup part of the workflow rather than a separate task.

Unifying runtime controls in a single platform

All six controls reinforce each other when they run through one evaluation layer.

The MCP integration

The Unleash MCP server connects an AI coding agent to all six controls. When Claude Code operates with the server configured, it gains access to three governance tools natively in its workflow:

• evaluate_change determines whether a flag is needed for a proposed change before code is written
• detect_flag prevents duplication by checking whether a matching flag already exists
• create_flag enforces naming conventions and inherited policy templates on any new flag the agent creates

Governance is built into the tools the agent calls. Integrating these capabilities takes one server configuration.

Scale and precedent

Governance must hold under load. A governance layer that degrades under traffic is a bottleneck teams route around, which means it stops governing.

Prompt-based defenses and text filtering are bypassed by adaptive attacks over 90 percent of the time. Governing the action path treats every tool call as a throttleable software capability. That holds even when agents face unexpected or malicious inputs. Six evaluated conditions between an agent and production, not six pages in a runbook.

Validating your runtime governance

When these six controls are wired in order, the outcome is an agent that ships continuously without requiring manual review for every change. Policy encoded once applies everywhere. Pauses are automatic and scoped. Audit records exist because the system ran, not because someone filled in a form.

There’s one test that confirms whether the controls are real: disable any one of them and watch whether the agent’s behavior changes. If it doesn’t, that control wasn’t running. Start wiring them with autonomous feature management, rollout policies first. A governance layer you can remove without consequence was never a governance layer at all.

FAQs about runtime governance for agentic AI

How does runtime governance differ from model-level AI safety?

Model safety focuses on pre-deployment alignment, bias testing, and prompt filtering to control what an AI says. Runtime governance manages what an agent does by intercepting execution trajectories, such as tool calls and system modifications, that emerge only during live operations.

What happens when an agent generates a flag that conflicts with an existing one?

Unleash prevents duplication and naming collisions through the MCP server integration, which allows agents to detect existing flags before creating new ones. If a conflict occurs at evaluation time, the platform applies inherited rollout policies to ensure the system defaults to a safe state.

How do I set approval thresholds without slowing down agent velocity?

Define high-risk action classes, such as database migrations or permission changes, upfront rather than reacting to individual incidents. In Unleash, you can automate these gates using change requests that pause execution only for specific risk categories, allowing lower-risk agent updates to ship without manual intervention.

Can this approach govern multiple agents modifying the same codebase?

Yes. The key is verifying the sequence of actions across different agents, not just whether each individual action is allowed. Unleash manages this at the flag layer by scoping targeting rules to specific agent identities, preventing one agent’s rollout from accidentally enabling features for another.

How do audit requirements for agentic AI differ in regulated industries?

Financial services and healthcare require semantic telemetry, which records the agent’s reasoning and intent behind an action rather than just the technical log. For these environments, that record has to be permanent and complete: Prudential, for example, built a custom ServiceNow integration that syncs every flag change and approval event into its compliance system of record.