New playbook: when your own AI model pays. Read

AI agent governance framework: decision rights, traces and evals

An AI agent governance framework gives each agent a narrow job, a named owner and explicit decision rights, logs every action, and gates every change with evals.

It sets which decisions each agent may make alone and records them. This page covers the CEO’s decisions, the autonomy options, cost drivers, and what Stripe, DoorDash, LG Uplus, Coinbase and Zurich built in 2026.

The yacht cut open, with the working deck and each winch highlighted.

What an AI agent governance framework covers

Four things: identity, decision rights, traceability and evaluation. Google’s October 2026 framing gives every agent its own identity stamped into logs, permissions approved by security, and one policy gateway for all traffic.

The Financial Stability Board’s October 2026 practices for financial institutions add two more: attribute every action and escalation to a specific agent, and test kill switches that range from a full stop to a graceful return to human operation.

Which agent decisions belong to the CEO

The CEO sets decision rights: what agents decide alone, what they recommend, and what stays human. IBM’s September 2026 rule is least privilege plus least agency. If the task only needs a recommendation, the agent may not need execution rights, and thresholds such as refund limits are enforced in the workflow itself.

The CEO also names a human owner for each agent’s outcomes. A Harvard Business School working paper on 202 enterprises (October 2026) found top-quartile firms run recurring production evaluations in 63% of cases against 4% in the bottom quartile, and counts a named human owner for agent outcomes among the practices it scores.

Agent autonomy levels compared

Companies give agents more autonomy one class of decision at a time. The JPMorgan treasury model (June 2026) moves from insight to signal to recommendation to execution.

LevelAgent doesHuman doesMove up when
ShadowRecommends on live cases with no write accessDecides and actsAgreement with human decisions is proven on real cases
Decision supportPrepares a decision-ready fileDecidesReviewers accept most recommendations unchanged
Constrained executionActs inside written thresholdsReviews exceptions and samplesEval scores hold and errors are reversible
Delegated authorityActs alone on a defined classAudits logs and outcomesThe class is low-risk and fully traced

What drives the cost of governing agents

Eval sets cost expert time: someone who knows the work must define what correct looks like. The second cost is reviewer time, because each human approval waits for a person. Shadow mode, described by Microsoft in July 2026, runs the agent on live cases without touching records or customers and gives the evidence needed to remove approvals safely.

What companies built in 2026

Stripe breaks each financial-compliance review into small pre-tested sub-tasks, runs agents on each, logs every tool output, and hands findings to human reviewers who decide. Stripe says median review handling time fell 26%, with over 96% helpfulness from reviewers (June 2026).

DoorDash built an evaluation harness for its ordering assistant with rubrics per agent, a conversation simulator and auto-grading of 2,000 live sessions a day. DoorDash says quality scores rose 8 points and the error rate nearly halved before nationwide launch (June 2026).

LG Uplus feeds production traces, user feedback and contact-centre experts’ judgments into evaluation datasets that drive every change to its contact-centre agents, Arize reported (August 2026).

Coinbase has a security agent assign each code change a risk level, and anything affecting the movement of money still gets thorough human testing and approval (August 2026). Cursor reports that agents create 75% of Coinbase’s pull requests (June 2026).

Zurich’s commercial-insurance agents turn broker submissions into decision-ready files across more than 20 countries. Cytora reports manual triage time down 80% (September 2026).

How to start with agent governance

Pick one agent already in use. Write its decision-rights map: what it decides, recommends and escalates. Give it an identity and a named owner, log every action, and build a replay set of fifty real cases before the next change ships.

What you build, decide and measure

What you build

  • Agents with one narrow job and an identity
  • A trace of every tool call and decision
  • Eval sets that gate every change

What the CEO decides

  • Which decisions agents may make alone, which they recommend, and which stay human
  • Who owns each agent’s outcomes by name
  • The quality bar an agent must pass before it reaches customers

What good looks like

  • Every agent action is attributed to a specific agent and a named human owner
  • No change reaches production without passing a replayed eval set
  • One switch stops any agent and returns its work to people

Other layers: Models and routing · Data and ontology · The learning loop

Cases on this layer

  • 26%

    shorter median compliance review time

    An agent system for financial-compliance investigations that breaks each review into a graph of small, pre-tested sub-tasks, quality-tests each, constrains agents with rails, and hands findings to human reviewers who decide (source).

  • 2,000

    sessions auto-graded a day before launch

    An evaluation harness for the multi-agent Ask DoorDash ordering assistant: rubrics per agent, a transcript builder, a conversation simulator and auto-grading of live sessions, used as the launch bar (source).

  • 70% → 96%

    agent accuracy through evaluation-driven development

    Evaluation-driven development for contact-centre agents on Arize AX: production traces, user feedback and domain-expert labels feed evaluation datasets that gate every change (source).

  • 75%

    of pull requests created by agents

    An agent-first development loop: agents turn Slack requests into pull requests and test builds, a security agent assigns each change a risk level, and simulator fleets verify behaviour against the spec before humans look (source).

  • 80%

    less manual submission triage time

    An agentic intake platform that reads broker submissions, digitises them and triages risks into a decision-ready workflow for underwriters across more than 20 countries (source).

  • 12,500

    banker hours saved in the first month of operation, July 2026

    Five specialist agents on Amazon Bedrock AgentCore that classify and read payslips and bank statements, run serviceability calculations and verify them against lending policy, then hand the banker a result to review for mortgage and credit-card applications (source).

  • $10M

    EBITDA uplift from two agent systems, Partners Group says

    Two agentic systems built on a cleaned proprietary policy-level dataset: one runs new-client policy processing, the other automates human-intensive policy checking; delivered by a seven-person team with Version 1 (source).

  • 2–3 minutes

    per ocean or air freight quote, down from one to three days

    Hundreds of in-house AI agents embedded across the shipment lifecycle (quoting, orders, appointments, missed-pickup checks) and now ocean and air quoting, ocean booking and customs brokerage, built and run under a lean waste-removal discipline by 450+ in-house engineers and data scientists (source).

Questions

What are AI agent evals?

Evals are repeatable tests that score an agent’s output against what an expert calls correct. A good eval set mixes replayed production cases with rubric scoring and runs before every change. DoorDash cut a six-hour manual regression test to about 20 minutes with its harness (June 2026), which made testing every change practical.

How much autonomy should an AI agent get?

Autonomy is set per class of decisions, based on results on real cases. Start in shadow mode, move to decision support, then to execution inside written thresholds. Spotify auto-merges a defined class of maintenance changes without human review while other changes keep review, which shows autonomy assigned per class (June 2026).

What is agent traceability?

Traceability means every action an agent takes is logged with the agent’s identity, its inputs, tool calls and outcome, so anyone can reconstruct why it acted. Stripe requires every tool output to be logged as an observation, giving audit trails that meet examination standards (June 2026).

Who should own AI agents in the organisation?

Each agent needs a named business owner accountable for its outcomes, plus a central function that runs identity, gateways and eval infrastructure. The October 2026 HBS working paper counts a named human owner for agent outcomes among the operating practices it scores. The business owner sets the quality bar, which the central team checks with evals.

We map how your company runs, pick the first processes with your leadership team, and put them on agents.