# Santiago Santa María Morales — EUREKA · Full knowledge corpus > Ideas, talks and analysis on applied AI, agentic AI, robotics, business and technology to improve organizations and lives. This file consolidates the full body of every knowledge unit and Enterprise AI pattern published on https://santismm.com, in English, each annotated with Evidence-First provenance (evidence level, confidence level and source types). It is intended for AI agents, RAG systems and deep-research crawlers that want the entire corpus in one fetch. The index version lives at https://santismm.com/llms.txt. Author: Santiago Santa María Morales — Head of AI Forward Deployed Engineers, LATAM (Google Cloud); physicist by training. Languages: English (/en), Español (/es), Português (/pt). Default: English. License: Content © Santiago Santa María Morales, licensed CC BY 4.0. Attribution required: credit the author and link the canonical URL. License-SPDX: CC-BY-4.0 License-URL: https://creativecommons.org/licenses/by/4.0/ Attribution: Santiago Santa María Morales — https://santismm.com Generated: 2026-08-01T23:51:29.224Z Evidence levels: production · simulation · benchmark · industry_observation · theoretical. Confidence levels: high · medium · low. Source types: personal_experience · production_system · benchmark · paper · industry_observation. ================================================================= # HARNESS ENGINEERING HANDBOOK ================================================================= ## HRN-001 — Harness Engineering: Definition and Overview URL: https://santismm.com/en/handbook/definition-and-overview | API: https://santismm.com/api/handbook/HRN-001 Category: Foundations | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-002, HRN-003, HRN-004 # Harness Engineering: Definition and Overview ## Executive Summary Harness Engineering is the discipline responsible for building reliable agentic systems for enterprise environments. A large language model is a probabilistic next-token predictor; an enterprise needs a dependable system that performs work, respects policy, and fails safely. The harness is everything engineered *around* the model — memory, tools, planning, orchestration, observability, evaluation, governance, and security — that closes the gap between the two. This chapter defines the discipline, states its thesis, and frames the rest of the handbook. ## Key Concepts - **Model:** The probabilistic core (an LLM or multimodal model) that maps a context to a distribution over next tokens. Powerful, but stateless, ungoverned, and non-deterministic by default. - **Harness:** The deterministic and semi-deterministic engineering scaffolding wrapped around one or more models to produce a dependable system. - **Agentic system:** A system in which a model drives a loop of perception, reasoning, and action against tools and an environment to pursue a goal. - **Reliability:** The probability that the system produces a correct, safe, and policy-compliant outcome under real conditions and load. - **Determinism boundary:** The deliberate line separating what the model is allowed to decide from what the harness fixes in code. - **Enterprise environment:** A setting with real stakes — regulated data, audit requirements, SLAs, and adversaries. ## Definition **Harness Engineering** is the engineering discipline concerned with the design, construction, and operation of the systems that surround probabilistic models so that the resulting agentic system is reliable, observable, governable, and secure enough for enterprise use. Where machine learning produces the *model*, Harness Engineering produces the *system*. Its unit of work is not a prompt or a weight matrix but the end-to-end loop that turns a goal into a verified, auditable outcome. ## Architecture Diagram ```mermaid flowchart TB subgraph Harness["The Harness (engineered scaffolding)"] direction TB PL[Planning & Goal Mgmt] OR[Orchestration] MEM[Memory] TL[Tools / Actuation] OBS[Observability] EVAL[Evaluation] GOV[Governance] SEC[Security] end USER([Goal / Request]) --> PL PL --> OR OR <--> MODEL{{Probabilistic Model}} OR <--> MEM OR <--> TL TL <--> ENV[(Enterprise Systems & Data)] OBS -.instruments.- OR EVAL -.scores.- OR GOV -.constrains.- OR SEC -.guards.- TL OR --> OUT([Verified, Auditable Outcome]) ``` ## Detailed Explanation The industry spent 2020–2023 learning that a better model is necessary but not sufficient. Demos that dazzle on a curated prompt collapse in production against ambiguous inputs, hostile users, stale data, partial tool failures, and the simple fact that the same input can yield a different output twice. The response was not "a smarter model" but *an engineered system around the model*. That system is the harness, and building it well is its own discipline. The central claim of this handbook is a **separation of concerns**: the model supplies open-ended reasoning and language; the harness supplies everything that makes that reasoning *dependable*. Treat the model as a brilliant, fast, and unreliable contractor. You would not hand such a contractor unmonitored access to production with no scope, no logging, no review, and no rollback. The harness is the scope, the logging, the review, and the rollback. **The model is not the system.** A useful mental model is to subtract the model and ask what remains. What remains is the harness, and it is where the overwhelming majority of enterprise engineering effort lives: - **Memory** decides what the model sees: what is retrieved, compressed, remembered, and forgotten (see HRN-005). - **Tools** are how the agent acts on the world, with typed contracts and failure semantics. - **Planning** decomposes goals and manages sub-goals and re-planning. - **Orchestration** runs the loop: who calls the model, with what context, and what happens to the output. - **Observability** makes every step a traceable, replayable span (see HRN-006). - **Evaluation** turns "it seems to work" into measured, regression-guarded quality (see HRN-007). - **Governance** encodes policy, approvals, and accountability as enforced controls. - **Security** treats the model as an untrusted, manipulable component and defends accordingly. These are not optional add-ons; they are the load-bearing structure. The taxonomy in HRN-003 makes the decomposition precise, and HRN-004 states the engineering principles that hold across all of them. **Why a new discipline?** Because the failure modes are new. Classical software is deterministic: given an input, it computes the same output, and you test it with assertions. Agentic systems are *stochastic and self-directed*: the same input may take different paths, invoke different tools, and reach different (sometimes wrong) conclusions. You cannot assert your way to confidence; you must *measure distributions*, bound the model's authority, and instrument everything. The skills required — probabilistic reliability, evaluation design, prompt-and-context engineering, tool contract design, and adversarial security — do not map cleanly onto either traditional ML or traditional backend engineering. That gap is the discipline. **Who it is for.** Harness Engineering is for the teams accountable for putting agents into production where it matters: platform engineers building agent runtimes, ML and applied-AI engineers shipping agentic features, security and governance functions who must sign off, and the architects who own the whole. It is explicitly *enterprise-first* — the constraints that define the discipline (audit, regulation, SLAs, adversaries, scale) are precisely the ones hobbyist tooling ignores. **An opinion, stated plainly:** the model is increasingly a commodity; the harness is the durable engineering asset and the moat. As frontier models converge and become swappable, the differentiated, defensible value of an enterprise AI system migrates into the harness — its memory architecture, its evaluation corpus, its governance controls, its observability. Investing in the harness is investing in the part that compounds. ## Observed Failure Modes - **Model-centric thinking:** Teams over-invest in prompt tweaking and model selection while under-investing in the harness, then blame the model for systemic failures. - **Demo-to-production cliff:** A system that works on happy-path demos has no memory discipline, no observability, and no evaluation, so it cannot survive contact with real load. - **Unbounded authority:** The model is allowed to decide things that should be fixed in deterministic code, producing unrecoverable or non-auditable actions. - **No measurement:** Without evaluation, regressions ship silently and "improvements" are vibes, not evidence. ## Cost Metrics The dominant cost driver in a naive system is model inference (tokens in/out). A well-engineered harness *reduces* this through memory compression, caching, routing cheap requests to cheap models, and short-circuiting with deterministic logic — while adding modest fixed costs for observability storage and evaluation runs. Mature harnesses typically shift spend from per-call inference toward amortized infrastructure, lowering cost per successful task even as per-request instrumentation grows. ## Scaling Characteristics The harness, not the model, determines how the system scales. Concurrency, statefulness of memory, orchestration fan-out, and tool back-pressure govern throughput and tail latency. Reliability tends to *degrade non-linearly* with task complexity (number of steps and tools), which is why the harness must be designed for graceful degradation rather than assuming a fixed success rate. ## Related Content - HRN-002 — A Brief History of Harness Engineering - HRN-003 — The Harness Taxonomy - HRN-004 — Harness Engineering Principles ## References - Industry observation on the "demo-to-production gap" in agentic systems (2023–2026). - Practitioner literature on agent architectures, tool use, and LLM orchestration frameworks. - Santa María, S. — Working notes on Harness Engineering as a discipline. ## FAQs **Q:** Is Harness Engineering just prompt engineering with a new name? **A:** No. Prompt engineering optimizes a single model interaction. Harness Engineering builds the whole reliable system around the model — memory, tools, orchestration, observability, evaluation, governance, and security. Prompting is one small input to one component. **Q:** If models keep getting better, won't the harness become unnecessary? **A:** The opposite. Better models raise the ceiling of what agents attempt, which increases the stakes and the surface area the harness must govern, observe, and secure. The harness is where enterprise reliability and differentiation live. **Q:** Where do I start? **A:** Read HRN-003 (the taxonomy) to map the components, then HRN-004 (principles). Begin instrumenting with observability (HRN-006) before optimizing anything — you cannot improve what you cannot measure. --- ## HRN-002 — A Brief History of Harness Engineering URL: https://santismm.com/en/handbook/a-brief-history-of-harness-engineering | API: https://santismm.com/api/handbook/HRN-002 Category: Foundations | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-001, HRN-003 # A Brief History of Harness Engineering ## Executive Summary Harness Engineering did not appear fully formed. It emerged over roughly four overlapping eras: prompt engineering, tool use, agents, and finally harnesses. Each era solved a problem and exposed the next one. This chapter traces that arc, names the inflection points, and explains why the accumulation of these lessons crystallized into a discipline whose unit of work is the whole system, not the prompt. ## Key Concepts - **Prompt engineering:** Shaping a single model interaction through instructions, examples, and formatting. - **Tool use (function calling):** Giving the model the ability to emit structured calls to external functions. - **Agent:** A model running a perceive–reason–act loop toward a goal, with memory and tools. - **Harness:** The full engineered scaffolding around the model that makes the agentic system reliable. - **Inflection point:** A moment where a prior abstraction stopped scaling and forced a new layer. ## Definition The **history of Harness Engineering** is the progression by which the locus of engineering effort moved outward from the prompt to the model interaction, then to the loop, and finally to the entire system surrounding the model — culminating in the recognition that building that system is a discipline in its own right. ## Architecture Diagram ```mermaid timeline title Eras of Harness Engineering Prompt Engineering : Single-shot instructions : Few-shot examples : Output formatting Tool Use : Function calling : Structured outputs : Retrieval (RAG) Agents : Reason-act loops : Multi-step planning : Working memory Harnesses : Orchestration & memory : Observability & eval : Governance & security ``` ## Detailed Explanation ### Era 1 — Prompt Engineering (the single interaction) The first wave treated the model as an oracle: craft the right prompt and read off the answer. Techniques accreted quickly — instructions, few-shot examples, role framing, chain-of-thought, and rigid output formatting. Prompt engineering was real and useful, but it optimized a *single* model call. Its ceiling was the moment a task required the model to *do* something in the world, or to remember anything beyond the context window. The lesson: a better prompt cannot make a stateless oracle into a system. ### Era 2 — Tool Use (the model acts and retrieves) The second wave gave the model hands. Function calling let the model emit structured requests that surrounding code executed — search, calculators, database queries, API calls. Retrieval-augmented generation (RAG) attacked the knowledge problem by fetching relevant context at query time instead of hoping it was memorized. This was a genuine architectural shift: now there was *code around the model* that mattered. But it was still largely a single hop — call the model, run a tool, return the result. Reliability problems emerged immediately: tools fail, return malformed data, time out, or are called with hallucinated arguments. The lesson: the moment the model touches real systems, you need contracts, validation, and failure handling — engineering, not prompting. ### Era 3 — Agents (the loop) The third wave closed the loop. Instead of one hop, the model ran iteratively: observe results, reason, act again, until the goal was met. Patterns like reason-and-act loops, tool-using planners, and multi-agent decompositions appeared, packaged in popular frameworks. Agents could now book a trip, refactor code, or triage a ticket across many steps. And here the *real* failure modes surfaced at scale: loops that never terminate, compounding errors where one bad step poisons the rest, runaway cost, context windows overflowing with accumulated history, and the impossibility of debugging a non-deterministic multi-step run after the fact. The agent frameworks made the loop easy to *write* and nearly impossible to *operate reliably*. The lesson: a loop without memory discipline, observability, evaluation, and bounded authority is a liability, not a product. ### Era 4 — Harnesses (the system) The fourth wave — where the discipline now lives — is the recognition that everything around the model *is the engineering problem*. Teams putting agents into enterprise production discovered they were spending almost all of their effort not on the model and not even on the agent loop, but on: - **Memory** that decides what the model sees and what it forgets (HRN-005); - **Observability** that turns an opaque run into traceable, replayable spans (HRN-006); - **Evaluation** that converts "seems fine" into measured, regression-guarded quality (HRN-007); - **Governance** that enforces policy and human approval as code; - **Security** that treats the model as an untrusted, prompt-injectable component; - **Orchestration** that bounds the loop, routes work, and degrades gracefully. This collection is the harness. Naming it mattered: it reframed "I built an agent" (a demo) into "I built a harness" (a system you can run in front of customers and auditors). HRN-003 formalizes the components as a taxonomy. ### Why the names changed Each rename reflected an expansion of the unit of accountability. Prompt → the call. Tool use → the call plus its actions. Agent → the loop. Harness → the system, including the parts no demo ever shows: what happens at 3 a.m. under load, under attack, under audit. The history is, in essence, the steady realization that the hard part was never the model. ## Production Evidence > **Evidence level:** theoretical · **Confidence:** medium · **Source:** industry_observation > > _Illustrative, representative narrative — not a single verified deployment._ - **Context:** Enterprise teams adopting LLM agents between 2023 and 2026. - **Scenario:** A team ships an impressive agent demo, then spends the following two quarters not improving the model but building memory management, tracing, evaluation harnesses, approval gates, and prompt-injection defenses to make it safe for production. - **Technology:** Frontier LLMs, function-calling APIs, vector stores, agent frameworks, tracing backends. - **Load:** From a handful of demo runs to sustained production traffic with adversarial users. - **Results:** Representative experience is that the harness, not the model, consumes the majority of engineering effort and is what ultimately gates the production launch. ## Observed Failure Modes - **Mistaking the era:** Treating a tool-use problem as a prompt problem, or an agent problem as a tool problem — applying yesterday's abstraction to today's failure. - **Framework lock-in as strategy:** Assuming an agent framework *is* the harness; frameworks provide the loop, not the observability, evaluation, governance, or security. - **Skipping straight to multi-agent:** Reaching for elaborate agent swarms before the single-agent harness is reliable, multiplying failure surface. ## Scaling Characteristics Each era pushed the reliability bottleneck outward. As systems scaled in steps and tools, the binding constraint moved from "is the prompt good" to "does the loop terminate, stay in budget, and remain auditable" — which is precisely the harness's domain. ## Related Content - HRN-001 — Harness Engineering: Definition and Overview - HRN-003 — The Harness Taxonomy ## References - Industry observation on the evolution of LLM application patterns, 2020–2026. - Practitioner literature on RAG, function calling, and agent loops. - Santa María, S. — Working notes on the emergence of Harness Engineering. ## FAQs **Q:** Did one product or paper invent Harness Engineering? **A:** No. It emerged from convergent practitioner experience across many teams hitting the same wall: agents are easy to demo and hard to operate. The discipline is a name for the lessons, not a single artifact. **Q:** Are the earlier eras obsolete? **A:** No — they are subsumed. Prompting, tool use, and agent loops are all components inside a modern harness. The harness adds the layers that make them dependable. **Q:** What comes after harnesses? **A:** Likely standardization and tooling maturity — shared harness platforms, interoperable observability and evaluation standards, and governance baked into runtimes — rather than a wholly new paradigm. The unit of accountability (the system) is now stable. --- ## HRN-003 — The Harness Taxonomy URL: https://santismm.com/en/handbook/the-harness-taxonomy | API: https://santismm.com/api/handbook/HRN-003 Category: Foundations | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-001, HRN-005, HRN-006, HRN-009, HRN-010 # The Harness Taxonomy ## Executive Summary The harness is not a monolith; it is a set of distinct components, each with a clear responsibility and clear interfaces to the others. This chapter provides the canonical taxonomy: eight components — memory, tools, planning, orchestration, observability, evaluation, governance, and security — organized into three layers (the execution loop, the cross-cutting concerns, and the controls). The taxonomy is the map the rest of the handbook fills in. ## Key Concepts - **Component:** A bounded part of the harness with a single primary responsibility. - **Execution layer:** Components that drive the perceive–reason–act loop (planning, orchestration, memory, tools). - **Cross-cutting layer:** Concerns that instrument or measure the loop without being inside it (observability, evaluation). - **Control layer:** Concerns that constrain what the loop is allowed to do (governance, security). - **Interface:** The contract by which two components exchange information or authority. ## Definition The **Harness Taxonomy** is the canonical decomposition of an agentic system's engineered scaffolding into named components and layers, defining each component's responsibility and its relationships to the others. It serves as a shared vocabulary and as a checklist: a production-grade harness must consciously address every component, even if it chooses a minimal implementation. ## Architecture Diagram ```mermaid flowchart TB subgraph CONTROL["Control Layer — constrains the loop"] GOV[Governance] SEC[Security] end subgraph CROSS["Cross-cutting Layer — measures the loop"] OBS[Observability] EVAL[Evaluation] end subgraph EXEC["Execution Layer — runs the loop"] PLAN[Planning & Goal Mgmt] ORCH[Orchestration] MEM[Memory] TOOL[Tools / Actuation] MODEL{{Model}} end PLAN --> ORCH ORCH <--> MODEL ORCH <--> MEM ORCH <--> TOOL OBS -. traces .-> EXEC EVAL -. scores .-> EXEC GOV -. policy gates .-> ORCH SEC -. guards .-> TOOL SEC -. sanitizes .-> MEM ``` ## Detailed Explanation The taxonomy organizes eight components into three layers. The layering matters: it tells you which components *do work*, which ones *watch the work*, and which ones *bound the work*. ### Execution Layer — runs the loop The components that actually produce the agent's behavior. - **Planning & Goal Management (HRN-009):** Decomposes a goal into sub-goals, decides the next action, manages re-planning when steps fail, and detects completion or impasse. Owns the question "what should happen next?" - **Orchestration (HRN-010):** The runtime that executes the loop — assembling context, calling the model, dispatching tool calls, handling retries and timeouts, routing between models or sub-agents, and enforcing budgets. Owns "who runs, with what, and what happens to the output." - **Memory (HRN-005):** Governs what enters the model's context: short-term working memory, long-term stores, retrieval, compression, and forgetting. Owns "what the model sees and remembers." - **Tools / Actuation:** The typed contracts through which the agent reads from and writes to enterprise systems, with explicit input validation, output schemas, idempotency, and failure semantics. Owns "how the agent affects the world." The model sits *inside* this layer as a called component, not as the system. This is the central reframing of HRN-001. ### Cross-cutting Layer — measures the loop These do not produce behavior; they make behavior visible and quantifiable. - **Observability (HRN-006):** Tracing, spans, structured logging, token/cost accounting, and replay. Turns an opaque non-deterministic run into an inspectable artifact. Owns "what happened, exactly?" - **Evaluation (HRN-007):** Offline and online measurement of quality — golden sets, LLM-as-judge, regression suites, task-completion metrics. Owns "is it actually good, and is it getting better or worse?" Observability and evaluation are co-dependent: evaluation needs the traces observability produces, and observability is most valuable when its data feeds evaluation. ### Control Layer — bounds the loop These constrain authority and defend the system. - **Governance (HRN-008):** Encodes policy, approval workflows, accountability, and auditability as enforced controls — human-in-the-loop gates, allowed-action policies, and records of who/what authorized each action. Owns "is this permitted, and who is accountable?" - **Security (HRN-011):** Treats the model and its inputs as untrusted: prompt-injection defense, tool sandboxing, least-privilege credentials, output validation, and data-exfiltration controls. Owns "can an adversary make this system do something it shouldn't?" ### How the components compose A request enters through **planning**, which hands a plan to **orchestration**. Orchestration assembles context from **memory**, calls the **model**, and routes the model's chosen actions to **tools**. Throughout, **observability** records every span and **evaluation** scores outcomes; **governance** gates risky actions and **security** guards the boundaries. The interfaces between components are where reliability is won or lost — a sloppy memory-to-orchestration contract or an unvalidated model-to-tool call is a classic source of production failure. ### Using the taxonomy The taxonomy is also a **maturity checklist**. For each component, ask: do we have it, is it explicit, and is it tested? Many "agent" projects implement only the execution layer and ship without observability, evaluation, governance, or security — the four components that distinguish a system from a demo. A balanced harness invests across all three layers. | Layer | Component | Primary responsibility | Chapter | |-------|-----------|------------------------|---------| | Execution | Planning & Goal Mgmt | Decide next action; re-plan | HRN-009 | | Execution | Orchestration | Run the loop; route; budget | HRN-010 | | Execution | Memory | Control context; retrieve; forget | HRN-005 | | Execution | Tools / Actuation | Act on the world via contracts | HRN-003 | | Cross-cutting | Observability | Trace, log, account, replay | HRN-006 | | Cross-cutting | Evaluation | Measure quality; guard regressions | HRN-007 | | Control | Governance | Enforce policy; approvals; audit | HRN-008 | | Control | Security | Defend against adversaries | HRN-011 | ## Observed Failure Modes - **Missing layers:** Implementing only the execution layer (a working loop) and omitting observability, evaluation, governance, and security — the demo-to-production cliff. - **Component coupling:** Blurring responsibilities (e.g., orchestration silently doing memory compression) so failures cannot be isolated or tested. - **Weak interfaces:** Unvalidated, untyped contracts between components, especially model→tool and retrieval→context, which propagate bad data through the loop. - **Over-orchestration:** Building elaborate multi-agent topologies before the single-agent components are individually reliable. ## KPIs | Metric | Target | Notes | |--------|--------|-------| | Component coverage | 8/8 addressed | Each taxonomy component consciously implemented or deliberately stubbed | | Interface validation rate | 100% of model→tool calls validated | Prevents propagation of malformed/hallucinated arguments | | Task completion rate | Domain-dependent | Measured by evaluation (HRN-007) | ## Cost Metrics Cost concentrates in the execution layer (inference and tool calls) and in observability storage (trace volume scales with steps). Evaluation adds periodic batch cost. Governance and security are mostly fixed engineering cost. A useful budgeting heuristic is to attribute cost per *component* so optimization targets the real driver rather than the most visible one. ## Scaling Characteristics Different components scale along different axes: orchestration scales with concurrency, memory with retained state and corpus size, observability with steps-per-run, and evaluation with corpus size and judge calls. Because the components scale independently, the taxonomy is also a capacity-planning tool — bottlenecks appear in specific components, not in "the agent" generically. ## Related Content - HRN-001 — Harness Engineering: Definition and Overview - HRN-005 — Memory in Agentic Systems - HRN-006 — Observability for Agentic Systems - HRN-009 — Planning and Goal Management - HRN-010 — Orchestration ## References - Practitioner literature on agent architectures and component decomposition. - Industry observation on production agent system structure, 2023–2026. - Santa María, S. — Working notes on the harness taxonomy. ## FAQs **Q:** Why eight components and not more or fewer? **A:** Eight is the minimal set that covers running the loop, measuring it, and bounding it without overlap. You can sub-divide (e.g., split tools from actuation) but the responsibilities remain. **Q:** Is the model a component of the harness? **A:** The model is *called by* the harness and sits inside the execution layer, but it is not itself part of the harness — the harness is precisely everything around it. **Q:** Can a small system skip the control layer? **A:** For a toy, yes; for an enterprise system, no. Governance and security are what make the system safe to put in front of customers, regulators, and adversaries. They can be minimal but must be conscious. --- ## HRN-004 — Harness Engineering Principles URL: https://santismm.com/en/handbook/harness-engineering-principles | API: https://santismm.com/api/handbook/HRN-004 Category: Foundations | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-001, HRN-003 # Harness Engineering Principles ## Executive Summary Components answer *what* a harness contains; principles answer *how* to build each one well. This chapter states the cross-cutting engineering principles of Harness Engineering — the rules that hold whether you are designing memory, orchestration, or a tool contract. They are opinionated by design: a principle that bends to every situation is not a principle. ## Key Concepts - **Principle:** A durable design rule that guides decisions across components. - **Determinism boundary:** The explicit line between model-decided and code-decided behavior. - **Evidence-first:** No claim of quality without measurement. - **Defense in depth:** Multiple independent layers so no single failure is catastrophic. - **Least authority:** Each component gets the minimum permission needed. - **Graceful degradation:** The system fails into a safe, reduced mode rather than collapsing. ## Definition The **Harness Engineering Principles** are a set of cross-cutting design rules that govern how the components of a harness are built and composed so that the resulting agentic system is reliable, observable, governable, and secure. They are the discipline's equivalent of the SOLID principles or the twelve-factor app — not a framework, but a stance. ## Architecture Diagram ```mermaid flowchart LR subgraph Principles P1[Reliability over Capability] P2[Determinism Boundaries] P3[Observability-First] P4[Evidence-First] P5[Defense in Depth] P6[Least Authority] P7[Graceful Degradation] P8[Idempotent Actuation] end P1 --> SYS[(Dependable Agentic System)] P2 --> SYS P3 --> SYS P4 --> SYS P5 --> SYS P6 --> SYS P7 --> SYS P8 --> SYS ``` ## Detailed Explanation ### 1. Reliability over capability The harness optimizes for the *floor* of behavior, not the ceiling. A system that is brilliant 95% of the time and catastrophic 5% of the time is, in an enterprise, a liability — the 5% is what makes the news and the audit. Prefer a narrower scope executed dependably to a broad scope executed erratically. Capability is the model's contribution; reliability is the harness's, and it is the one the enterprise is paying for. ### 2. Determinism boundaries Decide explicitly what the model is allowed to decide. Everything that *can* be deterministic *should* be: schema validation, routing, permission checks, retries, and post-conditions belong in code, not in a prompt. The model is reserved for the genuinely open-ended reasoning that only it can do. Drawing this boundary tightly is the single highest-leverage move in harness design — it shrinks the surface over which non-determinism can cause harm. ### 3. Observability-first Instrument before you optimize. You cannot debug, evaluate, or trust a non-deterministic multi-step system you cannot see. Every model call, tool invocation, and decision should be a structured, traceable, replayable span *before* the feature is considered complete (HRN-006). Observability is not a phase-two add-on; it is a precondition for every other principle, because each of them depends on measurement. ### 4. Evidence-first No quality claim ships without measurement. "It seems better" is not an engineering statement. Changes are gated by evaluation against golden sets and regression suites (HRN-007), and every consequential claim carries its provenance (the evidence model this very knowledge base uses). Evidence-first is what converts agent development from craft to engineering. ### 5. Defense in depth Assume any single layer will fail — the model will hallucinate, a tool will return garbage, a user will inject a malicious prompt — and ensure no single failure is catastrophic. Layer independent controls: input validation *and* output validation *and* permission gates *and* monitoring. The model is an untrusted component; treat its output as you would treat unvalidated user input (HRN-011). ### 6. Least authority Every component and tool receives the minimum authority required for its job and no more. Read-only by default; write access scoped and gated; destructive actions behind human approval (PAT-001-class controls). The blast radius of a compromised or confused agent is bounded by the authority you granted it — so grant little. ### 7. Graceful degradation When something fails, fail *into* a safe, reduced mode — escalate to a human, return a conservative answer, or decline — rather than crashing or, worse, taking a confident wrong action. The harness must have well-defined behavior for impasse, budget exhaustion, tool outage, and low confidence. A system that does not know how to give up safely is not production-ready. ### 8. Idempotent and reversible actuation Because the loop is stochastic and may retry, actions on the world should be idempotent where possible and reversible where not. A retried tool call must not double-charge a customer; a write should be safe to repeat; high-impact actions should be staged, confirmable, and rollback-capable. This principle is what makes retries — essential for reliability — safe. ### Tensions between principles The principles are not always aligned. Reliability-over-capability constrains what the model is allowed to attempt; observability-first adds latency and cost; least-authority slows development. Good harness engineering is the art of resolving these tensions *deliberately* and documenting the trade-off, rather than letting one principle silently win. The meta-principle: **make the trade-off explicit and measurable.** | Principle | Primary risk it mitigates | Main cost it imposes | |-----------|---------------------------|----------------------| | Reliability over capability | Catastrophic tail behavior | Reduced scope | | Determinism boundaries | Unbounded non-determinism | Up-front design effort | | Observability-first | Undebuggable runs | Storage, latency | | Evidence-first | Silent regressions | Eval infrastructure | | Defense in depth | Single-point catastrophe | Redundant controls | | Least authority | Large blast radius | Slower iteration | | Graceful degradation | Confident wrong actions | Extra fallback paths | | Idempotent actuation | Harmful retries | Action design complexity | ## Observed Failure Modes - **Principle theater:** Citing the principles in a design doc but not enforcing them in code or CI. - **Capability chasing:** Letting an impressive model capability widen scope past what the harness can reliably control. - **Optimizing the unseen:** Tuning prompts and chains before observability exists, so "improvements" are unmeasured. - **All-or-nothing failure:** No degraded mode, so any single component outage takes the whole system down or produces a confident error. ## Cost Metrics The principles trade marginal per-request cost (instrumentation, validation, redundant checks) for large reductions in the cost of failure (incidents, rework, audit findings, reputational damage). The economically correct framing is *expected cost including tail events*, where the principles consistently pay for themselves. ## Scaling Characteristics Principles compound at scale. Determinism boundaries and least authority bound the failure surface as step count and concurrency grow; observability- and evidence-first keep a growing system debuggable and regression-safe. Systems built without the principles tend to degrade super-linearly as they scale, because every new capability adds unbounded, unmeasured, over-privileged surface. ## Related Content - HRN-001 — Harness Engineering: Definition and Overview - HRN-003 — The Harness Taxonomy ## References - Analogy to established software principles (SOLID, twelve-factor, defense in depth) adapted to agentic systems. - Industry observation on agentic system reliability practices, 2023–2026. - Santa María, S. — Working notes on harness design principles. ## FAQs **Q:** Which principle matters most? **A:** Observability-first is the practical entry point because every other principle depends on measurement. Determinism boundaries is the highest-leverage design decision. They reinforce each other. **Q:** Aren't these just general software engineering principles? **A:** Several are adapted from classic engineering, which is intentional — agentic systems are still software. But the determinism boundary, evidence-first measurement of a stochastic system, and treating the model as untrusted input are specific to the harness. **Q:** How do I enforce principles, not just state them? **A:** Encode them in CI and runtime: schema validation as code, eval gates on merge, permission checks at the tool boundary, and required tracing. A principle that is not enforced is a wish. --- ## HRN-005 — Memory in Agentic Systems URL: https://santismm.com/en/handbook/memory-in-agentic-systems | API: https://santismm.com/api/handbook/HRN-005 Category: Memory | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-003, PAT-004, PAT-006 # Memory in Agentic Systems ## Executive Summary Memory is the harness component that decides what the model sees on every call and what persists across calls. Because the model is stateless and its context window is a hard, costly budget, memory is not a database feature bolted on — it is an active, opinionated curation system. This chapter covers the memory hierarchy (working, short-term, long-term), the context window as the binding constraint, and the three operations that make memory tractable at scale: retrieval, compression, and forgetting. ## Key Concepts - **Working memory:** The immediate context the model is reasoning over right now — the current step's assembled prompt. - **Short-term memory:** The accumulated state of the current task or session (conversation, intermediate results, scratchpad). - **Long-term memory:** Persistent knowledge across sessions — user preferences, prior outcomes, organizational facts. - **Context window:** The fixed token budget for a single model call; the scarcest resource in the loop. - **Retrieval:** Selecting relevant items from a larger store to place into context. - **Compression:** Reducing the token footprint of information while preserving its useful content (summarization, distillation). - **Forgetting:** Deliberately dropping or down-weighting information to control cost, relevance, and staleness. ## Definition **Memory in an agentic system** is the harness subsystem that manages the lifecycle of information the model uses — its acquisition, storage, selection into context, compression, and removal — across the time horizons of a single step, a single task, and the lifetime of the system. Its job is to put the *right* information in the model's limited context at the *right* time, and nothing else. ## Architecture Diagram ```mermaid flowchart TB subgraph LTM["Long-term Memory (persistent)"] VEC[(Vector / Semantic Store)] KV[(Structured Facts / Profiles)] EPI[(Episodic: past task outcomes)] end subgraph STM["Short-term Memory (per task)"] HIST[Conversation / Step History] SCR[Scratchpad / Intermediate Results] end RET[Retrieval] --> CTX COMP[Compression] --> CTX LTM --> RET STM --> COMP STM --> CTX CTX[[Working Memory = Assembled Context]] --> MODEL{{Model}} MODEL --> WRITE[Memory Writer] WRITE --> STM WRITE --> LTM FORGET[Forgetting / Eviction] -.prunes.-> STM FORGET -.prunes.-> LTM ``` ## Detailed Explanation ### The context window is a budget, not a container The single most important fact about memory is that the context window is *finite and expensive*, and quality degrades as you fill it. Even with large windows, packing everything in raises cost and latency and dilutes the model's attention on what matters (the "lost in the middle" effect). Memory engineering is therefore a *budgeting* problem: every token spent on history or retrieved context is a token not spent reasoning. The harness must continuously decide what earns its place in the window. ### The memory hierarchy - **Working memory** is whatever is in the context window for the current call. It is assembled fresh every step from the other tiers. - **Short-term memory** holds the evolving state of the current task: the conversation so far, tool results, and a scratchpad of intermediate reasoning. It grows monotonically unless managed, which is why it is the primary target for compression and forgetting. - **Long-term memory** persists across tasks and sessions: semantic stores (often vector-indexed for similarity retrieval), structured profiles and facts (key-value or relational), and episodic records of past task outcomes the agent can learn from. Long-term memory is what lets an agent be *consistent* across sessions and *improve* over time. ### Retrieval — choosing what to surface Retrieval selects relevant items from long-term (and sometimes short-term) memory to inject into working memory. The dominant approach is semantic similarity over embeddings, frequently augmented with keyword/lexical search (hybrid retrieval) and re-ranking. Retrieval quality dominates downstream answer quality: irrelevant or missing context cannot be fixed by a better prompt. Common refinements include query rewriting, metadata filtering, and recency/authority weighting. Patterns such as PAT-006 (knowledge retrieval) formalize these choices. ### Compression — fitting more into the budget When short-term memory outgrows the budget, the harness compresses it. Techniques range from simple truncation (drop oldest turns), to rolling summarization (replace old turns with a running summary), to hierarchical/semantic compression (summarize at multiple granularities and keep pointers to detail). Compression is lossy by definition, so the engineering question is *what is safe to lose* — and that is task-specific. A coding agent must keep exact identifiers; a support agent can summarize chit-chat aggressively. ### Forgetting — deliberate, not accidental Forgetting is an active control, not a bug. The harness must drop information that is stale (a fact that has changed), irrelevant (off-topic context), or out of budget (eviction under pressure). Without explicit forgetting, long-term memory accumulates contradictions and noise, and short-term memory overflows. Good forgetting policies down-weight by recency and relevance, expire facts with known volatility, and resolve conflicts (the newest authoritative value wins). Forgetting is also a *governance* surface: data-retention and right-to-be-forgotten requirements live here. ### Memory writing and consolidation Closing the loop, the harness decides what from a completed step or task to *write back* to memory: extract durable facts, summarize the episode, update the user profile. This consolidation step — analogous to moving working memory into long-term storage — is what turns a stateless model into a system that accrues knowledge. Done carelessly, it is also how an agent poisons its own future context with a hallucinated "fact," which is why writes should be validated like any other actuation. ### Memory as an attack surface Anything written to memory and later read into context is a prompt-injection vector. Retrieved documents and stored "facts" can carry adversarial instructions. Memory therefore intersects directly with security (HRN-011): treat retrieved and recalled content as untrusted input, not as trusted system prompt. ## Production Evidence > **Evidence level:** theoretical · **Confidence:** medium · **Source:** industry_observation > > _Illustrative, representative scenario — not a verified single deployment._ - **Context:** Long-running enterprise assistant agents (support, research, coding) operating over multi-turn sessions. - **Scenario:** Naive accumulation of full conversation history into the context window drives cost and latency up and answer quality down as sessions lengthen; introducing rolling summarization plus hybrid retrieval restores quality at a fraction of the token cost. - **Technology:** Frontier LLMs, vector store, hybrid retriever with re-ranking, summarization model for compression. - **Load:** Sessions ranging from a few turns to hundreds, with long-term stores from thousands to millions of items. - **Results:** Representative experience is a substantial reduction in tokens per turn and improved task consistency once memory is actively managed rather than passively accumulated. ## Observed Failure Modes - **Context overflow:** Unmanaged short-term memory exceeds the window, causing truncation of exactly the information that mattered. - **Lost in the middle:** Over-stuffed context degrades attention to mid-prompt content; more context yields worse answers. - **Retrieval miss:** The relevant document is never surfaced, and the model confidently answers from a gap. - **Stale or contradictory memory:** Long-term store holds outdated or conflicting facts; the agent acts on the wrong one. - **Memory poisoning:** A hallucinated or adversarial "fact" is written back and contaminates future reasoning. ## KPIs | Metric | Target | Notes | |--------|--------|-------| | Retrieval precision/recall | Domain-dependent, measured | Quality of items surfaced into context | | Context utilization | Below window with headroom | Tokens used vs. budget per call | | Tokens per turn | Minimized for quality held constant | Direct cost driver | | Answer groundedness | High | Share of claims supported by retrieved context | ## Cost Metrics Memory is a primary cost lever. Tokens placed in context are paid on every call, so compression and precise retrieval directly reduce inference spend. Long-term memory adds storage and embedding/indexing cost, and compression adds summarization-model calls. The net is almost always favorable: active memory management trades cheap batch summarization and indexing for expensive per-call context tokens. ## Scaling Characteristics Memory scales along two axes: session length (drives short-term memory and compression frequency) and corpus size (drives long-term store and retrieval latency). Retrieval latency and quality are the usual bottlenecks as the long-term store grows; sharding, filtering, and re-ranking become necessary. Crucially, well-managed memory keeps per-call cost *flat* as sessions lengthen, whereas naive accumulation makes it grow without bound. ## Related Content - HRN-003 — The Harness Taxonomy - PAT-004 — (memory / context pattern) - PAT-006 — (knowledge retrieval pattern) ## References - Research on context-length effects in LLMs ("lost in the middle"). - Practitioner literature on RAG, hybrid retrieval, and re-ranking. - Santa María, S. — Working notes on agent memory architecture. ## FAQs **Q:** With million-token context windows, is memory engineering obsolete? **A:** No. Larger windows raise the budget but do not remove it — cost, latency, and attention dilution still scale with what you put in. Bigger windows make memory engineering *more* valuable, not less, because the temptation to over-stuff is greater. **Q:** Is memory just RAG? **A:** Retrieval (RAG) is one operation within memory. Memory also covers short-term/working state, compression, forgetting, and write-back consolidation. RAG without those is incomplete. **Q:** Should the model decide what to remember? **A:** Partly. The model can propose what to consolidate, but the harness should validate and govern writes — unvalidated self-writes are how agents poison their own memory. --- ## HRN-006 — Observability for Agentic Systems URL: https://santismm.com/en/handbook/observability-for-agentic-systems | API: https://santismm.com/api/handbook/HRN-006 Category: Observability | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-003, HRN-007 # Observability for Agentic Systems ## Executive Summary Observability is the harness component that turns an opaque, non-deterministic agent run into an inspectable, replayable artifact. You cannot debug, evaluate, govern, or trust a multi-step stochastic system you cannot see — which is why observability is a precondition for nearly every other harness capability, not a phase-two add-on. This chapter covers traces and spans adapted for agents, token and cost accounting as first-class telemetry, evaluation hooks, and deterministic replay. ## Key Concepts - **Trace:** The complete record of a single agent run — every step from goal to outcome. - **Span:** A single unit of work within a trace (a model call, a tool invocation, a retrieval, a decision) with inputs, outputs, timing, and metadata. - **Token/cost accounting:** Per-span and per-trace tracking of tokens in/out and resulting cost. - **Evaluation hook:** An instrumentation point where evaluation logic can score a span or trace, online or offline. - **Replay:** Re-executing a recorded trace deterministically to reproduce and debug behavior. - **Cardinality:** The dimensionality of telemetry tags; high cardinality aids analysis but raises storage cost. ## Definition **Observability for agentic systems** is the harness subsystem that captures, structures, and stores a complete, queryable record of every agent run — its spans, inputs, outputs, model calls, tool calls, costs, and decisions — such that any run can be understood after the fact, compared across versions, scored by evaluation, and replayed deterministically. It answers the question "what, exactly, happened, and why?" ## Architecture Diagram ```mermaid flowchart TB RUN[Agent Run] --> TRACE[Trace] subgraph TRACE[Trace: one run] direction TB S1[Span: Plan] S2[Span: Model Call] S3[Span: Tool Call] S4[Span: Retrieval] S5[Span: Decision] end S2 --> TOK[Token / Cost Accounting] TRACE --> STORE[(Trace Store)] STORE --> QUERY[Query & Dashboards] STORE --> REPLAY[Deterministic Replay] STORE --> EVALH[Evaluation Hooks] EVALH --> EVAL[Evaluation HRN-007] QUERY --> ALERT[Alerting / Monitors] ``` ## Detailed Explanation ### Why classic observability is not enough Traditional APM assumes deterministic services: a request, a few synchronous calls, a response. Agentic systems break those assumptions. A single run may take a *different path each time*, fan out across many model and tool calls, loop an unknown number of times, and produce *natural-language* inputs and outputs that ordinary metrics cannot summarize. Observability for agents must therefore capture not just latency and errors but the *semantic content* of each step — the prompt sent, the completion returned, the tool arguments chosen, the reasoning. Without that content, a trace tells you *that* the agent failed but never *why*. ### Traces and spans, adapted for agents The trace/span model from distributed tracing is the right backbone, with agent-specific span types: - **Model-call spans** record the assembled prompt (or a reference to it), the completion, the model and parameters, token counts, and latency. - **Tool-call spans** record the tool, the (validated) arguments, the result or error, and retries. - **Retrieval spans** record the query, the items returned, and their scores — essential for diagnosing memory misses. - **Decision/plan spans** record the agent's choice of next action and, where available, its rationale. Spans nest to form the full causal tree of a run. The richer the captured content, the more debuggable the system — at the cost of storage and privacy exposure, which must be managed (redaction, sampling, retention). ### Token and cost accounting as first-class telemetry In agentic systems, *cost is a behavior*, not just a bill. A regression that causes an extra reasoning loop or a bloated context shows up first as a token spike. Observability must therefore treat token counts and derived cost as first-class metrics, attributed per span, per trace, per user, and per agent version. This makes cost regressions detectable, runaway loops alertable, and per-task economics measurable — closing the loop with the cost-metrics discipline that recurs across the handbook. ### Evaluation hooks Observability and evaluation (HRN-007) are co-dependent. Evaluation needs the traces; observability is most valuable when its data feeds scoring. The harness should expose *evaluation hooks* — instrumentation points where a scorer (a rule, a classifier, or an LLM-as-judge) can attach to a span or trace, either *online* (scoring live traffic for monitoring) or *offline* (replaying stored traces against a new model or prompt). Designing these hooks into the trace format from day one is what makes continuous evaluation cheap later. ### Deterministic replay The most powerful agent-specific capability is replay: re-running a recorded trace to reproduce its behavior. Because the model is non-deterministic, true replay requires capturing enough to *pin* the run — recorded model outputs (to replay without re-calling the model), tool results, retrieved context, and random seeds where applicable. Replay enables three things that are otherwise nearly impossible: reproducing a production failure locally, regression-testing a prompt or model change against real historical traffic, and A/B comparing two harness versions on identical inputs. A harness without replay debugs by guesswork. ### Privacy, redaction, and retention Capturing full prompts and completions means capturing potentially sensitive data. Observability must integrate redaction (PII scrubbing), access controls on the trace store, and retention policies — these are governance (HRN-008) and security (HRN-011) concerns that the observability layer enforces in practice. ## Production Evidence > **Evidence level:** theoretical · **Confidence:** medium · **Source:** industry_observation > > _Illustrative, representative scenario — not a verified single deployment._ - **Context:** Teams operating multi-step agents in production who initially shipped with only basic logging. - **Scenario:** An intermittent failure (the agent occasionally takes a wrong action) is undiagnosable from logs; after adding full trace/span capture with replay, the failing run is reproduced locally and traced to a retrieval miss that fed the model a misleading document. - **Technology:** Tracing backend with agent-aware span types, trace store, replay tooling, token/cost telemetry. - **Load:** Production traffic with long-tail, hard-to-reproduce failures. - **Results:** Representative experience is that mean-time-to-diagnosis drops sharply once runs are fully traced and replayable, and that cost regressions become visible the moment they occur. ## Observed Failure Modes - **Logs without structure:** Free-text logs that record *that* something happened but not the span tree, inputs, and outputs needed to understand it. - **No content capture:** Capturing latency and errors but not prompts/completions, leaving failures undiagnosable. - **Unbounded cardinality/storage:** Capturing everything at full fidelity for every run, exploding storage cost; needs sampling and retention policy. - **No replay:** Inability to reproduce non-deterministic failures, forcing debug-by-guesswork. - **Privacy leakage:** Capturing sensitive prompt content without redaction or access control. ## KPIs | Metric | Target | Notes | |--------|--------|-------| | Trace coverage | ~100% of runs traced | Every production run produces a trace | | Mean time to diagnosis | Minimized | Time from failure report to root cause via traces/replay | | Cost attribution coverage | Per span/trace/version | Enables cost-regression detection | | Replay fidelity | High | Share of recorded traces that replay deterministically | ## Cost Metrics Observability adds storage cost (proportional to traces × spans × captured content) and a small runtime overhead per span. Sampling, redaction, tiered retention, and storing references to large payloads control this. The cost is repaid by faster incident resolution and by *making token/cost itself observable*, which typically surfaces inference savings that dwarf the observability spend. ## Scaling Characteristics Trace volume scales with traffic × steps-per-run, so deep agentic workflows generate disproportionately more telemetry than shallow services. Storage and query cost are the scaling bottlenecks; head-based and tail-based sampling, aggregation, and retention tiers keep it bounded. Replay storage scales with the fidelity captured, trading storage for reproducibility. ## Related Content - HRN-003 — The Harness Taxonomy - HRN-007 — Evaluation of Agentic Systems ## References - Distributed tracing concepts (spans, traces) adapted to agentic workloads. - Practitioner literature on LLM observability and tracing tooling. - Santa María, S. — Working notes on agent observability and replay. ## FAQs **Q:** Isn't logging enough? **A:** No. Unstructured logs cannot reconstruct the causal span tree of a multi-step, branching run, and they rarely capture the semantic content (prompts, completions, retrieved context) needed to explain a failure. Structured traces with replay are required. **Q:** Why track cost in the observability layer? **A:** Because in agentic systems cost is a *behavior*: extra loops and bloated context show up as token spikes before they show up anywhere else. Cost telemetry is how you catch those regressions. **Q:** What is the single most valuable capability? **A:** Deterministic replay. It turns "we can't reproduce it" into a routine local debug session and enables regression-testing changes against real historical traffic. --- ## HRN-007 — Evaluation of Agentic Systems URL: https://santismm.com/en/handbook/evaluation-of-agentic-systems | API: https://santismm.com/api/handbook/HRN-007 Category: Evaluation | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-006, PAT-009, PAT-015 # Evaluation of Agentic Systems ## Executive Summary Evaluation is the harness component that converts "it seems to work" into a measured, defensible claim. Because agents are non-deterministic and operate over open-ended tasks, you cannot assert your way to confidence — you must measure distributions of behavior against known-good references and guard against regressions. This chapter covers offline and online evaluation, golden sets, LLM-as-judge, regression suites, and task-completion metrics, and argues that evaluation is the dividing line between agent *craft* and agent *engineering*. ## Key Concepts - **Offline evaluation:** Scoring an agent against a fixed dataset before deployment. - **Online evaluation:** Scoring live production traffic (with user signals or shadow judges). - **Golden set:** A curated dataset of inputs with known-good expected outputs or acceptance criteria. - **LLM-as-judge:** Using a model to score outputs against a rubric where exact-match is impossible. - **Regression suite:** A set of cases run on every change to catch quality drops. - **Task-completion rate:** The share of attempts that achieve the goal end-to-end. - **Trajectory evaluation:** Scoring the *path* an agent took, not only its final answer. ## Definition **Evaluation of an agentic system** is the harness subsystem that measures the quality, safety, and reliability of agent behavior against defined criteria — across curated datasets (offline) and live traffic (online) — and gates changes on the results. It answers two questions: "is it good enough to ship?" and "did this change make it better or worse?" ## Architecture Diagram ```mermaid flowchart TB subgraph OFFLINE["Offline Evaluation (pre-deploy)"] GOLD[(Golden Set)] --> RUNO[Run Agent] RUNO --> SCORE1[Scorers] SCORE1 --> GATE{Regression Gate} GATE -->|pass| SHIP[Deploy] GATE -->|fail| BLOCK[Block / Investigate] end subgraph ONLINE["Online Evaluation (live)"] PROD[Production Traffic] --> TRACE[Traces HRN-006] TRACE --> JUDGE[LLM-as-Judge / Rules] PROD --> USERSIG[User Signals] JUDGE --> MON[Monitors & Dashboards] USERSIG --> MON end subgraph SCORERS["Scorer Types"] EXACT[Exact / Rule-based] LLMJ[LLM-as-Judge] HUMAN[Human Review] end SCORE1 --- SCORERS JUDGE --- SCORERS MON -.feeds new cases.-> GOLD ``` ## Detailed Explanation ### Why agent evaluation is hard Three properties make this harder than traditional software testing. First, **non-determinism**: the same input can yield different outputs and even different *paths*, so a single pass/fail assertion is meaningless — you measure rates over runs. Second, **open-endedness**: many correct answers exist, so exact-match scoring fails and you need rubric-based or semantic judgment. Third, **multi-step trajectories**: an agent can reach a right answer via a wrong (unsafe, expensive) path, so evaluating only the final output is insufficient. Evaluation design is the art of turning these properties into measurable signals. ### Offline evaluation and golden sets Offline evaluation runs the agent over a fixed **golden set** — curated inputs paired with expected outputs or acceptance criteria — before anything ships. The golden set is the most valuable asset evaluation produces; it encodes what "good" means for your domain and compounds over time. Build it from real (anonymized) production cases, known failure cases, and edge cases, and *grow it from every incident*: when the agent fails in production, the fix is not just a code change but a new golden case so the failure can never silently return. This is the regression discipline (PAT-015-class knowledge validation) that makes the system improvable. ### Scorer types — matching the method to the task - **Exact / rule-based scorers** for tasks with verifiable outputs (a correct SQL result, a valid JSON schema, a passing unit test). Cheap, deterministic, trustworthy — use them wherever possible. - **LLM-as-judge** for open-ended outputs where exact-match fails (summaries, explanations, plans). A model scores against a rubric. Powerful but fallible: judges have biases (position, verbosity, self-preference), so calibrate them against human labels, use clear rubrics, and prefer pairwise comparison over absolute scoring where feasible. Treat the judge as an *instrument that itself needs evaluation*. - **Human review** for the highest-stakes or most-ambiguous cases, and to calibrate the automated scorers. Expensive, so reserve it for the cases that need it and for keeping the cheaper scorers honest. ### Task-completion and trajectory metrics The headline metric for an agent is usually **task-completion rate**: end-to-end, did it achieve the goal? Beneath it sit step-level and trajectory metrics — did it choose appropriate tools, avoid unnecessary steps, stay in budget, and avoid unsafe actions along the way? Trajectory evaluation catches agents that are "right for the wrong reasons," which is precisely the kind of fragility that breaks under distribution shift. Pair completion rate with cost-per-task and safety-violation rate to avoid optimizing one at the expense of the others. ### Online evaluation Offline tells you about your dataset; only **online evaluation** tells you about reality. Online evaluation scores live traffic using implicit user signals (acceptance, edits, escalations, retries), shadow LLM-judges running on production traces (HRN-006), and periodic human audits of sampled runs. Online evaluation is also how new golden cases are discovered — production is the richest source of the edge cases your offline set is missing. The loop is: observe (HRN-006) → judge online → harvest failures into the golden set → guard with offline regression. ### Regression gating — evaluation as a CI gate The discipline becomes engineering when evaluation *gates changes*. Every prompt edit, model swap, or tool change runs against the regression suite, and a quality drop blocks the merge — exactly as a failing unit test blocks code. This is the operational form of the evidence-first principle (HRN-004): no change ships on vibes. Because agent evaluation is rate-based and partly LLM-judged, gates use thresholds and statistical comparison rather than a single boolean, but the principle is identical. ## Production Evidence > **Evidence level:** theoretical · **Confidence:** medium · **Source:** industry_observation > > _Illustrative, representative scenario — not a verified single deployment._ - **Context:** Teams iterating on a production agent by frequently changing prompts and swapping models. - **Scenario:** Without an evaluation gate, a prompt change that improved one case silently regressed several others, shipping a net-worse agent; introducing a golden-set regression suite with LLM-as-judge plus rule-based scorers caught the regression before deploy. - **Technology:** Golden-set harness, rule-based and LLM-as-judge scorers, CI gate, online judge over production traces. - **Load:** Frequent changes against a golden set ranging from dozens to thousands of cases. - **Results:** Representative experience is that quality stops drifting once changes are gated, and that the golden set — continuously grown from production failures — becomes the team's most valuable asset. ## Observed Failure Modes - **Vibes-based shipping:** Changes evaluated by spot-checking a few prompts, so regressions ship unnoticed. - **Overfitting the golden set:** Tuning until the fixed set passes while real-world quality stalls — mitigated by growing the set from fresh production cases. - **Naive LLM-as-judge:** Trusting an uncalibrated judge with known biases; treating its scores as ground truth without validation against humans. - **Final-answer-only scoring:** Missing agents that reach right answers via unsafe or expensive paths. - **No online evaluation:** Strong offline numbers that do not survive contact with real, shifting traffic. ## KPIs | Metric | Target | Notes | |--------|--------|-------| | Task completion rate | Domain-dependent, trended | Primary headline quality metric | | Regression suite pass rate | 100% before deploy | Gate on every change | | Judge–human agreement | High, calibrated | Validates the LLM-as-judge instrument | | Safety-violation rate | Near zero | Trajectory-level, not just final answer | | Cost per successful task | Minimized | Pairs with completion to prevent over-optimization | ## Cost Metrics Evaluation adds cost in three places: running the agent over the golden set (inference), LLM-as-judge scoring (more inference), and human review (labor). These are controlled by tiering — cheap rule-based scorers first, LLM-judge for the open-ended subset, humans for calibration and high-stakes cases. The cost is repaid by preventing regressions, which are far more expensive once shipped. Reusing observability traces (HRN-006) for offline replay avoids re-running the model where possible. ## Scaling Characteristics Evaluation cost scales with golden-set size × scorer cost × change frequency. As the set grows, sampling and tiered scoring keep regression runs affordable; the most informative cases can be weighted or run more often. Online evaluation scales with sampled traffic rather than all of it. The golden set itself scales *up* in value as it grows — the opposite of most cost curves — because each added case is a permanently guarded failure mode. ## Related Content - HRN-006 — Observability for Agentic Systems - PAT-009 — (evaluation / judging pattern) - PAT-015 — Knowledge Validation ## References - Practitioner literature on LLM evaluation, LLM-as-judge calibration, and golden datasets. - Industry observation on regression gating for agentic systems, 2023–2026. - Santa María, S. — Working notes on agent evaluation discipline. ## FAQs **Q:** Can I just trust an LLM-as-judge? **A:** Use it, but treat it as an instrument that itself needs evaluation. Calibrate it against human labels, give it explicit rubrics, prefer pairwise comparison, and watch for known biases (position, verbosity, self-preference). **Q:** Where do golden cases come from? **A:** Real (anonymized) production traffic, known failure cases, and edge cases — and crucially, every production incident should add a new golden case so the failure cannot silently recur. **Q:** Offline or online evaluation — which do I need? **A:** Both. Offline gates changes before deploy against a known set; online tells you what is actually happening in reality and feeds new cases back into the offline set. They form a loop with observability. --- ## HRN-008 — Governance within the Harness URL: https://santismm.com/en/handbook/governance-within-the-harness | API: https://santismm.com/api/handbook/HRN-008 Category: Governance | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-003, GOV-001, PAT-001 # Governance within the Harness ## Executive Summary Governance is not a document that lives in a wiki — in a reliable agentic system it is a **runtime layer of the harness**. This chapter argues that enterprise AI obligations (regulatory, contractual, and risk-based) must be compiled into executable controls that sit on the critical path between the model's intent and the system's action. Harness Engineering treats governance as code: policy decision points, approval gates, and guardrails that observe, allow, transform, or block every tool call. Without this layer, an agent's autonomy is ungoverned by construction; with it, autonomy becomes bounded, auditable, and defensible. ## Key Concepts - **Policy Enforcement Point (PEP):** the harness component that intercepts an agent action and queries a decision. - **Policy Decision Point (PDP):** the engine that evaluates policy against the action's context and returns allow/deny/transform. - **Guardrail:** a runtime check on inputs or outputs (content, schema, PII, jurisdiction) that constrains behavior. - **Approval gate:** a control that suspends execution pending a human or higher-authority decision (see PAT-001). - **Policy-as-code:** governance rules expressed in a declarative, version-controlled, testable format. - **Audit trail:** the immutable record of what was attempted, what was decided, and why. ## Definition > **Governance within the harness** is the discipline of embedding policy enforcement, approval workflows, and guardrails as a first-class runtime layer of an agentic system, such that every model-initiated action is mediated by an explicit, auditable decision derived from enterprise policy. ## Architecture Diagram ```mermaid flowchart LR M[Model / Reasoning Loop] -->|proposed action| PEP[Policy Enforcement Point] PEP -->|context + action| PDP[Policy Decision Point] G[(Policy-as-code Bundle)] --> PDP PDP -->|allow| TOOL[Tool / Effector] PDP -->|transform| RW[Redact / Constrain] --> TOOL PDP -->|deny| BLK[Block + Explain] PDP -->|escalate| APR[Approval Gate / Human] APR -->|approved| TOOL APR -->|rejected| BLK PEP --> AUD[(Immutable Audit Log)] PDP --> AUD APR --> AUD TOOL --> AUD ``` ## Detailed Explanation The governance layer is structured around the classic **PEP/PDP split** borrowed from authorization architecture (XACML, OPA), adapted for non-deterministic agents. The enforcement point is woven into the harness's tool-invocation path so that *no* effectful action — sending an email, writing to a database, transferring funds, calling an external API — reaches an effector without first being evaluated. The decision point evaluates the action against a **policy bundle**: a versioned, testable set of rules covering who the agent acts for, what data classes it touches, which jurisdictions apply, and what spend or blast-radius limits are in force. Three enforcement outcomes matter beyond simple allow/deny. **Transform** lets the harness permit an action while neutralizing risk — redacting PII before an outbound call, downscoping a query, or capping a transaction amount. **Escalate** routes the action to an approval gate (PAT-001), suspending the agent's plan durably until a human or supervisor agent decides. **Deny-with-explanation** returns a structured rationale into the agent's context so the reasoning loop can replan rather than blindly retry. Guardrails operate at two boundaries. *Input guardrails* screen retrieved content and user instructions for injection, jailbreak patterns, and out-of-scope requests before they influence the plan. *Output guardrails* validate generated content and structured tool arguments against schema, content policy, and data-loss rules before they leave the trust boundary. Crucially, guardrails are **layered, not singular**: a single classifier is a single point of failure, so defense-in-depth combines deterministic checks (regex, schema, allowlists), statistical checks (classifiers), and model-based checks (LLM-as-judge) with conservative fail-closed defaults for high-risk actions. Governance also defines the **autonomy gradient**. The harness assigns each action class a control mode along a spectrum: fully autonomous, autonomous-with-logging, human-in-the-loop (approval required), or human-on-the-loop (human can interrupt). This mapping is itself policy: a refund under $50 may be autonomous; a refund over $5,000, or any action touching regulated data, demands an approval gate. The taxonomy of these control modes connects directly to the harness taxonomy (HRN-003) and to the enterprise governance framework (GOV-001), which supplies the obligations this layer compiles. Finally, governance is only credible if it is **observable and provable**. Every decision — the action proposed, the policy version consulted, the inputs, the verdict, and the rationale — is written to an immutable, queryable audit trail. This is what converts "we have an AI policy" into "we can demonstrate, per action, that policy was enforced," which is the evidentiary bar regulators and auditors actually apply. ## Production Evidence > **Illustrative / representative scenario.** Evidence level: theoretical · Confidence: medium · Source: industry_observation, personal_experience. The figures below are realistic ranges drawn from observed patterns, not measurements from a single verified deployment. - **Context:** A financial-services back-office agent that drafts and executes customer remediations. - **Scenario:** The agent must autonomously resolve low-value disputes while never autonomously moving funds above a threshold or touching another customer's data. - **Technology:** Orchestrator with a PEP on every tool call; OPA-style policy bundle; classifier + schema + allowlist guardrails; durable approval queue. - **Load:** Tens of thousands of actions/day; a single-digit percentage routed to approval gates. - **Results (representative):** In illustrative deployments of this shape, governance layers commonly reduce high-severity policy violations by an order of magnitude versus an ungoverned baseline, at the cost of added per-action latency in the low tens of milliseconds for deterministic checks and added end-to-end latency for escalated actions bounded by human response time. ### Lessons Learned Fail-closed defaults on high-risk action classes are non-negotiable; the expensive failures come from actions that were *never evaluated* because a new tool was added without a corresponding policy. Governance must therefore gate **tool registration**, not just tool invocation. ## Observed Failure Modes | Failure Mode | Trigger | Mitigation | |---|---|---| | Policy bypass | New tool added without a PEP hook | Gate tool registration; deny-by-default for unmapped actions | | Guardrail evasion | Prompt injection rewrites intent past a single classifier | Layered, fail-closed guardrails; input + output checks | | Approval fatigue | Over-broad gates flood humans, who rubber-stamp | Risk-tier gates; auto-approve low-risk with logging | | Stale policy | Policy bundle drifts from regulation | Version + test policy as code; periodic conformance review | | Silent transform | Redaction corrupts a legitimate action | Log transforms; surface rationale into agent context | | Audit gaps | Decisions not persisted before action executes | Write-ahead audit; deny if audit sink unavailable | ## KPIs | Metric | Target | Notes | |---|---|---| | Policy coverage (action classes mapped) | 100% | Unmapped → deny-by-default | | High-severity violation rate | → 0 | Per 10k actions | | Approval gate precision | High | Fraction of escalations that were warranted | | Decision latency (p95) | < 50 ms deterministic | Excludes human approval wait | | Audit completeness | 100% | Every effectful action has a decision record | | Mean time to policy update | Low (hours) | Policy-as-code CI/CD | ## Cost Metrics - **Per-action governance overhead:** deterministic checks add negligible compute; model-based guardrails add one or more auxiliary inference calls — budget for them in cost-per-task. - **Human approval cost:** the dominant variable cost; minimized by precise risk-tiering so only warranted actions escalate. - **Engineering cost:** policy authoring and conformance testing; amortized as reusable policy bundles across agents. ## Scaling Characteristics Deterministic enforcement scales horizontally and statelessly with the orchestrator. Model-based guardrails scale with inference capacity and are the throughput bottleneck under high action volume — cache and short-circuit them with cheap deterministic checks first. Approval gates scale with human capacity, not compute, so the design goal is to keep the escalated fraction small and stable as action volume grows. ## Related Content - HRN-003 — Taxonomy of harness layers and control modes. - GOV-001 — Enterprise AI Governance Framework (the obligations this layer enforces). - PAT-001 — Human Approval pattern (the approval-gate mechanism). ## References - NIST AI Risk Management Framework (AI RMF 1.0). - ISO/IEC 42001:2023 — AI management systems. - OASIS XACML and the PEP/PDP authorization model. - Open Policy Agent (OPA) — policy-as-code engine. ## FAQs **Q: Why not handle governance in the prompt?** A: Prompt instructions are advisory and defeasible by injection; harness-level enforcement is mandatory and auditable. Governance must be outside the model's persuadable surface. **Q: Doesn't gating every action add too much latency?** A: Deterministic checks cost single-digit-to-tens of milliseconds. Only escalated actions incur human-scale delay, and those are deliberately rare. **Q: How is this different from GOV-001?** A: GOV-001 defines the obligations and framework; HRN-008 is how those obligations are compiled into runtime controls inside the harness. --- ## HRN-009 — Planning and Goal Management URL: https://santismm.com/en/handbook/planning-and-goal-management | API: https://santismm.com/api/handbook/HRN-009 Category: Planning | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-003, HRN-010, PAT-010 # Planning and Goal Management ## Executive Summary A reliable agent does not merely react token-by-token — it pursues a goal through a represented, inspectable plan. Planning and goal management is the harness layer that turns a high-level objective into a structured, executable plan, tracks its state across long-horizon execution, and revises it when reality diverges from expectation. This chapter treats the plan as a **first-class data structure** owned by the harness, not an ephemeral chain-of-thought trapped in the context window. Externalizing the plan is what makes agent behavior auditable, resumable, and recoverable. ## Key Concepts - **Goal:** the desired end-state the agent is commissioned to achieve, with success criteria. - **Plan:** an ordered or partially-ordered set of tasks expected to achieve the goal. - **Task / step:** an atomic unit of work mapped to one or more tool calls. - **Decomposition:** the act of breaking a goal into tasks (see PAT-010). - **Plan representation:** the explicit structure (list, tree, DAG, state machine) the plan is stored in. - **Replanning:** revising the plan in response to failure, new information, or changed constraints. - **Plan state:** the durable record of which steps are pending, in-progress, done, or failed. ## Definition > **Planning and goal management** is the harness discipline of representing a goal and its decomposed plan as explicit, durable state; selecting and sequencing tasks against that plan; and continuously reconciling the plan with observed outcomes through replanning. ## Architecture Diagram ```mermaid flowchart TD GOAL[Goal + Success Criteria] --> DEC[Decomposition] DEC --> PLAN[(Plan as DAG\npersisted state)] PLAN --> SEL[Task Selector] SEL --> EXE[Execute Step\nvia Orchestrator] EXE --> OBS[Observe Outcome] OBS -->|success| UPD[Update Plan State] OBS -->|failure / new info| REP[Replan] REP --> PLAN UPD --> DONE{Goal\nsatisfied?} DONE -->|no| SEL DONE -->|yes| END[Report + Verify] UPD --> PLAN ``` ## Detailed Explanation Planning begins with **decomposition**: converting a goal into tasks whose completion, in aggregate, satisfies the goal's success criteria (PAT-010 names this pattern). Decomposition strategies trade off cost against adaptivity. *Plan-then-execute* commits a full plan up front — cheap, predictable, and easy to govern, but brittle when the world surprises it. *Interleaved planning* (the ReAct family) plans one step at a time from observations — adaptive and robust, but more expensive and harder to bound. *Hierarchical planning* combines them: a coarse plan of phases, each expanded just-in-time into concrete steps. Mature harnesses choose per-task: deterministic, well-understood workflows favor plan-then-execute; open-ended research favors interleaving. The **plan representation** is the load-bearing decision. A flat task list suffices for linear work; a **DAG** captures dependencies and unlocks parallelism (the orchestrator, HRN-010, can dispatch independent branches concurrently); a **state machine** is right when transitions are governed and must be exhaustively enumerable. Whatever the shape, the plan must be *externalized and persisted*. A plan living only inside the model's context is lost on a crash, invisible to observability, and impossible to govern. Externalizing it lets the harness resume from the last durable state, lets humans inspect and edit it, and lets the governance layer (HRN-008) reason about what the agent intends to do *before* it does it. **Goal management** is the layer above individual plans. It tracks success criteria explicitly, so completion is *verified* rather than asserted by the model. It manages sub-goals and their dependencies, handles goal conflicts and prioritization, and enforces termination conditions — step budgets, time budgets, and cost ceilings — that prevent the classic failure of an agent looping forever on an unachievable goal. Termination criteria are part of the goal specification, not an afterthought. **Replanning** is where planning earns its place in a *reliable* system. The world is non-stationary: tools fail, data is stale, assumptions break. The replanning loop watches each step's outcome against expectation and triggers revision on divergence — a failed step, a precondition that no longer holds, or new information that invalidates downstream tasks. Effective replanning is *scoped*: prefer local repair (retry, substitute a tool, insert a recovery step) over discarding the whole plan, and escalate to full re-decomposition only when local repair fails repeatedly. This connects directly to recovery strategy patterns and keeps replanning from thrashing. A replan budget — a cap on how many times a plan may be revised before escalating to a human or supervisor — prevents the agent from burning cost in a planning loop. ## Production Evidence > **Illustrative / representative scenario.** Evidence level: theoretical · Confidence: medium · Source: industry_observation, personal_experience. Ranges below are representative of observed patterns, not measurements from one verified system. - **Context:** A multi-step data-migration agent reconciling records across systems. - **Scenario:** The goal ("migrate and reconcile account X") decomposes into extract, transform, validate, and load phases with inter-step dependencies. - **Technology:** DAG plan persisted to a durable store; interleaved replanning on validation failures; step + cost budgets. - **Load:** Long-horizon tasks spanning minutes to hours, dozens of steps each. - **Results (representative):** Externalizing the plan and adding scoped replanning typically lifts task-completion rate substantially over a plan-once baseline on tasks with realistic failure rates, primarily by recovering from transient failures locally instead of aborting the whole run. ### Lessons Learned The biggest reliability gains come not from smarter initial plans but from **cheap, well-scoped replanning** and **explicit termination budgets**. Unbounded agents fail by looping; bounded agents fail safely and escalate. ## Observed Failure Modes | Failure Mode | Trigger | Mitigation | |---|---|---| | Plan loss on crash | Plan held only in context | Persist plan as durable state | | Infinite loop | No termination budget | Step/time/cost budgets + replan cap | | Over-decomposition | Goal split into trivial micro-steps | Right-size step granularity to tool calls | | Replan thrash | Full re-decomposition on every minor failure | Scoped local repair before global replan | | Unverified completion | Model asserts done without checking criteria | Explicit success-criteria verification | | Dependency violation | Step run before its precondition holds | DAG ordering enforced by orchestrator | ## KPIs | Metric | Target | Notes | |---|---|---| | Task completion rate | High | Goal-level, criteria-verified | | Steps per task | Minimal | Lower is cheaper; watch over-decomposition | | Replan rate | Low–moderate | Spikes signal brittle plans or flaky tools | | Plan recovery success | High | Fraction of failures repaired without abort | | Budget breach rate | → 0 | Tasks hitting step/cost ceilings | ## Cost Metrics - **Cost per task** scales with steps × per-step inference + tool cost; over-decomposition inflates it directly. - **Planning overhead:** interleaved planning adds an inference call per step; plan-then-execute amortizes one planning call across many steps. - **Replan cost:** each replan is additional inference; the replan cap bounds worst-case cost per task. ## Scaling Characteristics DAG plans scale execution throughput by exposing parallelizable branches to the orchestrator. Plan complexity scales the planning-inference cost super-linearly, so hierarchical decomposition (plan phases coarsely, expand lazily) keeps per-task planning cost bounded as goals grow. Durable plan state scales with the number of concurrent in-flight goals, not their length, making the state store the component to size for concurrency. ## Related Content - HRN-003 — Where planning sits in the harness taxonomy. - HRN-010 — Orchestration executes the plan and dispatches parallel branches. - PAT-010 — Goal Decomposition pattern. ## References - Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models." - Wang et al., "Plan-and-Solve Prompting." - Classical AI planning: STRIPS / HTN (hierarchical task network) planning literature. ## FAQs **Q: Should the plan live in the context window?** A: No. Persist it as durable state. Context is volatile, size-limited, and ungovernable; a persisted plan is resumable, inspectable, and auditable. **Q: Plan-then-execute or interleaved?** A: Choose per task. Deterministic workflows favor plan-then-execute; open-ended or failure-prone tasks favor interleaved replanning. Hierarchical planning blends both. **Q: How do I stop an agent from looping forever?** A: Make termination criteria part of the goal: step, time, and cost budgets plus a replan cap, with escalation to a human or supervisor on breach. --- ## HRN-010 — Orchestration URL: https://santismm.com/en/handbook/orchestration | API: https://santismm.com/api/handbook/HRN-010 Category: Orchestration | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-003, HRN-009, PAT-002, PAT-005 # Orchestration ## Executive Summary Orchestration is the engine room of the harness: the layer that decides *who acts, in what order, and what happens when a step fails*. It spans the spectrum from a single agent running a loop to fleets of specialized agents coordinated by a supervisor. This chapter frames orchestration as the bridge between a represented plan (HRN-009) and reliable execution, and argues that the central engineering problem is not intelligence but **durability**: long-running, non-deterministic, partially-failing workflows must survive crashes, resume cleanly, and never silently lose or duplicate effects. The right default is the simplest topology that meets the requirement — complexity in orchestration is a cost, not a virtue. ## Key Concepts - **Topology:** the arrangement of agents — single, pipeline, supervisor/worker, or network. - **Supervisor / orchestrator agent:** an agent that plans and delegates to workers (see PAT-002). - **Worker agent:** a specialized agent that executes a delegated sub-task (see PAT-005). - **Routing:** selecting the next agent, tool, or branch based on state. - **State machine:** an explicit graph of states and transitions governing execution. - **Durable execution:** workflow semantics where progress is checkpointed and resumable. - **Handoff:** transferring control and context from one agent to another. ## Definition > **Orchestration** is the harness discipline of executing a plan across one or more agents and tools — selecting topology, routing control, coordinating state, and guaranteeing durable, exactly-the-right-number-of-times execution under failure. ## Architecture Diagram ```mermaid flowchart TD subgraph Durable Workflow Engine SUP[Supervisor Agent] -->|delegate| R{Router} R -->|task A| W1[Worker: Retrieval] R -->|task B| W2[Worker: Code/Tool] R -->|task C| W3[Worker: Drafting] W1 --> AGG[Aggregator / Reducer] W2 --> AGG W3 --> AGG AGG --> SUP end SUP -->|checkpoint| ST[(Durable State Store)] ST -->|resume after crash| SUP SUP --> OUT[Verified Result] ``` ## Detailed Explanation **Topology selection** is the first and most consequential decision. A *single agent* with tools is the correct default for most tasks: it is cheapest, easiest to observe, and has the fewest coordination failure modes. Reach for multi-agent only when the task genuinely benefits — when sub-tasks need *different* tool permissions, *different* context windows, or *parallel* independent execution. The common topologies are: **pipeline** (fixed sequence of stages), **supervisor/worker** (PAT-002 + PAT-005: a planner delegates to specialists and aggregates), and **network/peer** (agents hand off freely). Coordination cost rises sharply with topology freedom; peer networks are powerful but hardest to make reliable, govern, and debug. **Routing** is how control moves through the system. Routing can be *model-driven* (the supervisor chooses the next worker via tool-calling), *rule-driven* (deterministic transitions in a state machine), or *hybrid*. Deterministic routing is preferred wherever the path is known, because it is governable and testable; model-driven routing is reserved for genuinely open-ended branching. Encoding the workflow as an explicit **state machine** — states, allowed transitions, and guards — is the single highest-leverage reliability technique in orchestration: it bounds the space of behaviors, makes the system inspectable, and lets governance (HRN-008) attach controls to transitions. **Durability** is the property that separates a demo from a production system. Agentic workflows are long-running (seconds to hours), call flaky external tools, and may crash mid-flight. A durable execution engine checkpoints progress after each step so that on failure the workflow *resumes* from the last completed step rather than restarting. This demands careful effect semantics: tool calls with side effects must be **idempotent** or guarded by dedup keys so a resume does not double-charge a card or re-send an email. The hard cases are the *non-idempotent external effects*; the harness handles them with the saga pattern — record intent, execute, confirm, and provide compensating actions for partial failure. **State and context management** across agents is where multi-agent systems leak reliability. Each handoff (PAT-005) must transfer *exactly* the context the worker needs — too little and it fails, too much and it is expensive and prone to distraction. Shared state belongs in a durable store with clear ownership, not in a free-floating shared context window. Aggregation of worker outputs needs an explicit reducer with conflict resolution, because parallel workers will produce overlapping or contradictory results. Finally, orchestration owns **concurrency and failure isolation**. Parallel branches (exposed by the DAG plan from HRN-009) improve latency but require backpressure, rate-limit coordination across shared tools, and bulkheading so one failing worker cannot exhaust the budget or block siblings. Timeouts, circuit breakers, and per-worker budgets are orchestration concerns, not application concerns. ## Production Evidence > **Illustrative / representative scenario.** Evidence level: theoretical · Confidence: medium · Source: industry_observation, personal_experience. The numbers below are representative ranges, not a measurement from one verified deployment. - **Context:** A research-and-synthesis agent answering complex enterprise questions. - **Scenario:** A supervisor decomposes a question, dispatches parallel retrieval/analysis workers, and aggregates a cited answer. - **Technology:** Durable workflow engine, supervisor/worker topology, deterministic router for known stages, dedup keys on side-effecting tools. - **Load:** Concurrent multi-worker runs; each run minutes long with several external tool calls. - **Results (representative):** Parallel fan-out commonly cuts wall-clock latency by a meaningful multiple over sequential execution, while durable checkpointing reduces failed-run rates by eliminating crash-induced full restarts. The cost is higher token spend (more agents, more context) and added coordination complexity. ### Lessons Learned Most teams reach for multi-agent too early. The reliable progression is: make a single agent work, encode it as a state machine, add durability, *then* split into workers only where parallelism or permission isolation pays for the coordination cost. ## Observed Failure Modes | Failure Mode | Trigger | Mitigation | |---|---|---| | Duplicate side effects | Resume re-runs a non-idempotent step | Idempotency keys / saga compensation | | Lost progress on crash | No checkpointing | Durable execution engine | | Context loss at handoff | Worker under-receives state | Explicit, typed handoff contracts | | Coordination deadlock | Workers wait on each other | Acyclic routing, timeouts, supervisor arbitration | | Cost explosion | Recursive/peer delegation unbounded | Per-run agent budget + delegation depth cap | | Conflicting aggregation | Parallel workers disagree | Explicit reducer with conflict resolution | | Shared-tool throttling | Workers hammer one rate-limited API | Centralized rate-limit + backpressure | ## KPIs | Metric | Target | Notes | |---|---|---| | Task completion rate | High | End-to-end, verified | | Latency p50/p95/p99 | Minimized | Parallelism improves p50; tails dominated by slow workers | | Resume success rate | → 100% | Workflows that recover after a crash | | Duplicate-effect rate | → 0 | Idempotency correctness | | Cost per task | Bounded | Caps on agents/depth/tokens | | Throughput | Scales with concurrency | Limited by shared-tool rate limits | ## Cost Metrics - **Token cost** grows with agent count and per-agent context; multi-agent is materially more expensive than single-agent for the same task. - **Orchestration overhead:** supervisor planning + aggregation inference per run. - **Durability overhead:** checkpoint writes (cheap) vs. the large savings from not restarting failed runs. ## Scaling Characteristics Single-agent throughput scales horizontally and statelessly. Supervisor/worker scales sub-tasks in parallel up to shared-tool rate limits, which become the true ceiling. Durable workflow engines scale with the number of in-flight workflows; checkpoint storage and the dispatcher are the components to size. Peer/network topologies scale worst — coordination overhead and failure surface grow super-linearly with agent count, which is why bounded supervisor topologies are the enterprise default. ## Related Content - HRN-003 — Orchestration's place in the harness taxonomy. - HRN-009 — The plan that orchestration executes. - PAT-002 — Supervisor Agent pattern. - PAT-005 — Multi-Agent Delegation pattern. ## References - Temporal / durable-execution workflow engines (Saga pattern, workflow durability). - Anthropic, "Building Effective Agents" (single-agent-first, topology guidance). - LangGraph and state-machine orchestration for agents. ## FAQs **Q: Single agent or multi-agent?** A: Default to single agent. Add agents only for parallelism or permission/context isolation that pays back the coordination cost. **Q: Why a state machine instead of free-form agent loops?** A: State machines bound behavior, are testable, and let governance attach controls to transitions. Free-form loops are powerful but hard to make reliable or auditable. **Q: How do I avoid double-charging a customer on retry?** A: Make side-effecting tool calls idempotent (dedup keys) or wrap them in a saga with compensating actions, and run on a durable engine that resumes rather than restarts. --- ## HRN-011 — Security for Agentic Systems URL: https://santismm.com/en/handbook/security-for-agentic-systems | API: https://santismm.com/api/handbook/HRN-011 Category: Security | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: HRN-003, GOV-001, PAT-007 # Security for Agentic Systems ## Executive Summary Agentic systems expand the attack surface in a way traditional applications do not: the agent reads untrusted data, makes consequential decisions, and holds privileges to act — so a single compromised input can become a compromised action. Security for agentic systems is the harness layer that assumes the model can and will be manipulated, and engineers the surrounding scaffolding so that manipulation cannot cause harm. The governing principle is **least privilege with a constrained blast radius**: treat the model as an untrusted component, place security controls *outside* its persuadable surface, and ensure that even a fully hijacked agent can do only bounded, auditable damage. ## Key Concepts - **Prompt injection:** untrusted content that hijacks the agent's instructions or goals. - **Indirect prompt injection:** injection delivered via data the agent retrieves (documents, web pages, tool outputs). - **Tool sandboxing:** isolating tool execution so it cannot exceed its intended authority. - **Least privilege:** granting each agent the minimum permissions needed for its task. - **Agent identity:** a distinct, attributable principal for each agent, scoped and revocable. - **Data exfiltration:** unauthorized egress of sensitive data through tool outputs or rendered content. - **Lethal trifecta:** the dangerous combination of access to private data, exposure to untrusted content, and the ability to externally communicate. ## Definition > **Security for agentic systems** is the harness discipline of treating the model as an untrusted, manipulable component and engineering identity, permission, isolation, and egress controls so that the maximum harm any compromised agent can cause is bounded, attributable, and auditable. ## Architecture Diagram ```mermaid flowchart LR UNT[Untrusted Inputs\nuser, web, docs, tool output] --> IG[Input Guardrails\ninjection detection] IG --> AGENT[Agent Reasoning Loop\n(treated as untrusted)] AGENT -->|proposed tool call| AUTH[AuthZ + Least Privilege\nagent identity, scoped creds] AUTH --> SBX[Tool Sandbox\nisolation, allowlist] SBX --> EFF[Effector / External System] EFF --> OG[Output / Egress Guardrails\nDLP, exfil checks] OG --> SINK[Allowed Destination] AUTH -.deny/quarantine.-> BLK[Block + Alert] AGENT --> AUD[(Immutable Audit Log)] AUTH --> AUD OG --> AUD ``` ## Detailed Explanation The foundational threat is **prompt injection**, and the foundational mistake is trying to solve it inside the model. No amount of system-prompt hardening reliably stops a sufficiently clever instruction embedded in retrieved content, because the model has no robust, principled boundary between "data" and "instruction." Indirect injection is the dangerous variant: an agent that summarizes a web page or reads a ticket can be commandeered by text the attacker planted there. The harness response is **architectural, not prompt-based**: assume injection will sometimes succeed, and ensure that a hijacked agent still cannot do anything its *permissions* forbid. Input guardrails (injection/jailbreak detection) reduce the *frequency* of successful injection; permission and egress controls bound the *consequence*. Both are required; neither alone suffices. **Least privilege and agent identity** are the spine of agentic security. Each agent should run as a distinct, attributable principal with credentials scoped to exactly the resources its task requires — short-lived tokens, narrow OAuth scopes, read-only where writes aren't needed, and per-tenant data isolation enforced *below* the agent (in the data layer), never by asking the model nicely to stay in its lane. When an agent acts on behalf of a user, it should carry that user's authorization, not a god-mode service account, so the agent can never exceed what the user could do directly. Credentials must be injected by the harness at call time, never placed in the context window where injection could read and exfiltrate them. **Tool sandboxing** isolates execution. Code-running tools execute in ephemeral, network-restricted, resource-capped sandboxes. Tool catalogs are **allowlisted** per agent, so a hijacked agent cannot reach a tool it was never granted. High-consequence tools sit behind human approval (PAT-007 / PAT-001) so that even an authorized-but-manipulated call requires a human to commit it. The principle is *defense in depth*: AuthZ decides *whether* a call is permitted, the sandbox bounds *what the call can touch*, and the approval gate adds a human checkpoint for irreversible actions. **Data exfiltration** is the most under-appreciated agentic risk. The "lethal trifecta" — an agent with (1) access to private data, (2) exposure to untrusted content, and (3) the ability to communicate externally — is exploitable: injected instructions tell the agent to embed secrets into an outbound request, a rendered image URL, or a tool argument. The harness breaks the trifecta by removing at least one leg for sensitive contexts: restrict egress destinations to an allowlist, run data-loss-prevention checks on every outbound payload, strip or pin external content rendering, and forbid the agent from constructing arbitrary outbound URLs. If an agent must touch private data, its ability to communicate externally must be tightly constrained, and vice versa. Underpinning all of it is **observability and auditability** (HRN-006): every action, the identity that performed it, the permission decision, and the egress check must be logged immutably. Security you cannot prove is security you do not have. These controls implement the obligations defined in the governance framework (GOV-001) and slot into the broader harness taxonomy (HRN-003). ## Production Evidence > **Illustrative / representative scenario.** Evidence level: theoretical · Confidence: medium · Source: industry_observation, personal_experience. The descriptions below are representative attack/mitigation patterns, not measurements from one verified deployment. - **Context:** A customer-support agent with access to a knowledge base and the ability to email customers. - **Scenario:** An attacker plants injection text in a support ticket attempting to make the agent email another customer's account data to an external address. - **Technology:** Per-agent scoped credentials, egress allowlist, DLP on outbound mail, injection detection on retrieved content, audit log. - **Load:** High ticket volume; a small but nonzero fraction carry injection attempts. - **Results (representative):** In deployments of this shape, the architectural controls (egress allowlist + DLP + least privilege) block the *consequence* of injection even when detection misses the *attempt*, reducing successful exfiltration toward zero while injection-detection alone leaves residual risk. ### Lessons Learned Detection-only strategies fail eventually; the reliable defenses are the architectural ones that bound consequence. Breaking the lethal trifecta — especially constraining egress — does more for safety than any single classifier. ## Observed Failure Modes | Failure Mode | Trigger | Mitigation | |---|---|---| | Direct prompt injection | Malicious user instruction | Input guardrails + permission bounding | | Indirect injection | Malicious text in retrieved data | Treat all retrieved content as untrusted; egress controls | | Data exfiltration | Lethal trifecta exploited | Break the trifecta: egress allowlist + DLP | | Privilege escalation | Over-broad service credentials | Per-agent scoped, short-lived creds; user-delegated authZ | | Credential leakage | Secrets in context window | Inject creds at call time, never in context | | Sandbox escape | Unrestricted code/network in tools | Network-isolated, resource-capped ephemeral sandboxes | | Confused deputy | Agent misused as a proxy for forbidden actions | Carry caller identity; authZ at the effector | ## KPIs | Metric | Target | Notes | |---|---|---| | Successful exfiltration rate | → 0 | The metric that matters most | | Injection detection recall | High | Reduces attempt frequency, not the sole defense | | Permission scope tightness | Minimal grants | Audit for unused/over-broad permissions | | Egress allowlist coverage | 100% | No arbitrary outbound destinations | | Mean time to revoke | Low | Compromised agent identity revocation | | Audit completeness | 100% | Every action attributable | ## Cost Metrics - **Guardrail inference cost:** injection/DLP classifiers add auxiliary inference per request — budget in cost-per-task. - **Sandbox overhead:** ephemeral sandbox spin-up adds latency to code tools; amortize with warm pools. - **Engineering cost:** scoped identity and egress allowlisting are upfront IAM work that pays back across all agents. ## Scaling Characteristics Permission and identity controls scale with the IAM/secrets infrastructure, statelessly per call. Guardrail classifiers scale with inference capacity and are the throughput cost; short-circuit with cheap deterministic checks (allowlists, regex, schema) first. Sandboxes scale with a managed pool; warm pools trade idle cost for latency. Egress controls scale trivially and should never be the bottleneck — they are the cheapest high-value control in the stack. ## Related Content - HRN-003 — Security's place in the harness taxonomy. - GOV-001 — Governance obligations that security controls implement. - PAT-007 — Tool/permission control pattern (sandboxing and gated tool use). ## References - OWASP Top 10 for LLM Applications (LLM01 Prompt Injection, LLM06 Sensitive Information Disclosure). - Simon Willison, "The lethal trifecta for AI agents." - NIST AI RMF and NIST SP 800-53 (least privilege, identity). - MITRE ATLAS — adversarial threat landscape for AI systems. ## FAQs **Q: Can prompt injection be fully prevented?** A: No. Engineer for it: assume injection succeeds sometimes and bound the consequence with least privilege, egress control, and sandboxing. **Q: What is the single highest-value control?** A: Breaking the lethal trifecta — most cheaply by constraining egress to an allowlist with DLP — so a hijacked agent cannot exfiltrate data. **Q: Should agents share a service account?** A: No. Give each agent a distinct, scoped, short-lived identity, and have it carry the calling user's authorization so it can never exceed the user's own rights. --- ## HRN-012 — Case Studies in Harness Engineering URL: https://santismm.com/en/handbook/case-studies-in-harness-engineering | API: https://santismm.com/api/handbook/HRN-012 Category: Case Studies | Updated: 2026-06-21 Evidence: theoretical | Confidence: medium | Source: industry_observation, personal_experience | Status: Draft Related: ARCH-001, ARCH-002, HRN-001 # Case Studies in Harness Engineering ## Executive Summary This chapter grounds the abstract harness layers in three end-to-end stories. Each is a **representative, anonymized composite** — synthesized from common patterns across the industry, not an account of a single named deployment, and not a source of verified metrics. The point is to show how the layers interact under load: how memory, planning, orchestration, governance, security, and observability stop being separate chapters and become one system. Read together, the cases reinforce the thesis that reliability in enterprise agents is an engineering property of the harness, not an emergent property of the model. ## Key Concepts - **Composite case study:** an illustrative scenario assembled from recurring real-world patterns, explicitly not a verified single deployment. - **End-to-end:** spanning ingestion of intent through verified, governed action and observation. - **Harness layer interaction:** how memory, planning, orchestration, governance, security, and observability compose. ## Definition > A **harness engineering case study** is a structured narrative that traces a goal through every layer of an agentic system to expose the design decisions, failure modes, and trade-offs that determine reliability. ## Architecture Diagram ```mermaid flowchart TD INTENT[User Intent] --> PLAN[Planning HRN-009] PLAN --> ORC[Orchestration HRN-010] ORC --> MEM[(Memory HRN-005)] ORC --> GOV[Governance HRN-008] GOV --> SEC[Security HRN-011] SEC --> TOOLS[Tools / Effectors] TOOLS --> OBS[Observability HRN-006] OBS --> EVAL[Evaluation HRN-007] EVAL -.feedback.-> PLAN MEM -.context.-> PLAN ``` ## Detailed Explanation ### Case Study 1 — Financial Operations: the Reconciliation Agent > Representative composite. No verified metrics; ranges are illustrative. **Goal.** Autonomously reconcile daily transactions across two ledgers and remediate discrepancies under a strict spend authority. **Harness design.** Planning (HRN-009) decomposes the goal into a DAG: extract, match, classify discrepancies, remediate, report. Orchestration (HRN-010) runs it on a **durable workflow engine** so an overnight crash resumes from the last checkpoint rather than restarting — critical because some remediation steps move money and must never double-execute (idempotency keys + saga compensation). Governance (HRN-008) places an **approval gate** on any remediation above a threshold; below it, the agent acts autonomously with full audit logging. Memory (HRN-005) holds reconciliation rules and prior-resolution precedents. Observability (HRN-006) traces every match decision. **Outcome (illustrative).** The agent clears the long tail of trivial discrepancies autonomously and escalates the consequential ones, shifting human effort from *doing* reconciliation to *approving* exceptions. **Lesson:** durability + idempotency were the load-bearing decisions; the "intelligence" was the easy part. ### Case Study 2 — Customer Support: the Resolution Agent > Representative composite. No verified metrics; ranges are illustrative. **Goal.** Resolve inbound support tickets end-to-end — answer questions, update accounts, issue small credits — while never leaking one customer's data to another and never being hijacked by ticket content. **Harness design.** This is a **security-first** harness (HRN-011). Each agent instance carries the *requesting customer's* authorization, so data isolation is enforced below the model, not by prompting. Retrieved knowledge-base and ticket content is treated as untrusted; **egress is allowlisted** and outbound messages pass DLP — breaking the lethal trifecta even when injection detection misses an attempt. A single-agent topology (HRN-010) keeps it simple; a reflection step (PAT-003-style self-check) reviews the drafted reply before send. Governance gates credits above a small threshold to a human. **Outcome (illustrative).** Most tickets resolve without human touch; injection attempts in tickets fail to cause harm because the *consequence* is bounded by permissions and egress control, not merely by detection. **Lesson:** architectural security beat classifier security; the win came from constraining what a hijacked agent *could* do. ### Case Study 3 — Knowledge Work: the Research-and-Synthesis Agent > Representative composite. No verified metrics; ranges are illustrative. **Goal.** Answer complex internal questions with cited, trustworthy synthesis over a large corpus. **Harness design.** A **supervisor/worker** topology (HRN-010, PAT-002 + PAT-005): the supervisor decomposes the question and dispatches parallel retrieval and analysis workers, then an aggregator reconciles their findings into a cited answer. Memory (HRN-005) supplies retrieval context; planning (HRN-009) is interleaved because the path depends on what early retrieval surfaces. Evaluation (HRN-007) runs an LLM-as-judge groundedness check that *fails the answer if claims aren't cited*, feeding back into a replan. Observability traces the fan-out so cost and latency per worker are visible. **Outcome (illustrative).** Parallel fan-out improves latency over sequential research at the cost of higher token spend; the groundedness gate is what makes the output trustworthy enough to ship. **Lesson:** multi-agent earned its complexity here specifically because of *parallelism* and the need to verify before answering — not because multi-agent is inherently better. ### Cross-cutting observations Across all three, the same truths recur: (1) the simplest topology that meets the requirement wins; (2) durability and idempotency, not cleverness, decide whether a long-running agent is production-grade; (3) governance and security are *runtime* layers, not documents; (4) verification (evaluation) before action is what converts plausible output into trustworthy output. These connect to the reference architectures (ARCH-001, ARCH-002) and restate the core thesis of HRN-001: reliability is engineered into the harness. ## Production Evidence > **Illustrative / representative scenarios.** Evidence level: theoretical · Confidence: medium · Source: industry_observation, personal_experience. All three case studies are anonymized composites assembled from recurring patterns. They contain no measurements from any single verified production deployment, and any quantities are illustrative ranges. - **Context:** Financial operations, customer support, and enterprise knowledge work. - **Scenario:** End-to-end agentic automation under real enterprise constraints (spend authority, data isolation, citation trust). - **Technology:** Durable workflow engines, scoped agent identity, egress allowlists, supervisor/worker orchestration, LLM-as-judge evaluation. - **Results:** Directional and qualitative; presented to illustrate design trade-offs, not to assert benchmarked outcomes. ### Lessons Learned The recurring lesson is restraint: teams that succeeded added complexity (multi-agent, autonomy) *only* where a specific requirement justified it, and invested early in the unglamorous layers — durability, identity, audit — that determine whether anything works in production. ## Observed Failure Modes | Case | Dominant Failure Mode | Decisive Mitigation | |---|---|---| | Reconciliation | Double-executing a money-moving step on resume | Idempotency keys + saga compensation | | Support | Data exfiltration via injected ticket content | User-delegated authZ + egress allowlist + DLP | | Research | Unsupported claims presented as fact | Groundedness eval gate before answer | ## KPIs | Metric | Reconciliation | Support | Research | |---|---|---|---| | Task completion rate | High (with escalation) | High | High | | Human-touch rate | Low (exceptions only) | Low | Moderate (review) | | Safety incident rate | → 0 (gated spend) | → 0 (bounded blast radius) | → 0 (cited only) | | Latency | Batch-tolerant | Interactive | Improved by fan-out | | Cost per task | Low | Low | Higher (multi-agent) | ## Cost Metrics - **Reconciliation:** cheap per task (single agent, deterministic); dominant cost is human approval of exceptions. - **Support:** cheap per task; guardrail/DLP inference is the marginal add. - **Research:** highest per task due to multi-agent token spend; justified by parallel latency and verified quality. ## Scaling Characteristics The single-agent cases (reconciliation, support) scale horizontally and cheaply, bounded by external-tool rate limits and human approval capacity. The multi-agent research case scales sub-tasks in parallel up to shared-retrieval rate limits, with token cost growing per added worker — the classic latency-vs-cost trade that defines when multi-agent is worth it. ## Related Content - ARCH-001 — Reference architecture exemplifying single-agent durable workflows. - ARCH-002 — Reference architecture exemplifying supervisor/worker orchestration. - HRN-001 — Definition and Overview (the thesis these cases reinforce). ## References - Anthropic, "Building Effective Agents" and multi-agent research-system writeups. - Industry post-mortems and architecture writeups on durable agentic workflows. - The harness chapters HRN-005 through HRN-011, which these cases compose. ## FAQs **Q: Are these real deployments?** A: No. They are anonymized composites assembled from recurring industry patterns, presented to illustrate design trade-offs. They contain no verified production metrics. **Q: What is the single most transferable lesson?** A: Add complexity only where a requirement demands it, and invest first in durability, identity, and audit — the layers that decide whether an agent survives production. **Q: Why include multi-agent only in case 3?** A: Because that is the only case where parallelism and verification justified the coordination cost. The others are deliberately single-agent. --- ## HRN-013 — Glossary URL: https://santismm.com/en/handbook/glossary | API: https://santismm.com/api/handbook/HRN-013 Category: Reference | Updated: 2026-06-21 Evidence: theoretical | Confidence: high | Source: industry_observation | Status: Draft Related: HRN-001 # Glossary ## Executive Summary This glossary is the canonical reference for terms used across the Harness Engineering handbook and the wider Santismm Knowledge Platform. Definitions are crisp, machine-extractable, and intended to be cited directly. Where a term has a dedicated chapter, the entry points to it. ## Definition The following are the canonical definitions of terms used throughout this corpus. Terms are grouped for readability; within groups they are roughly ordered from foundational to specialized. ### Core concepts - **Agent:** a system that uses a language model in a loop to pursue a goal, deciding which actions (tool calls) to take based on observations until a stopping condition is met. - **Agentic system:** a software system whose behavior is driven by one or more agents, including all the surrounding scaffolding required to make them reliable. - **Harness:** the engineered scaffolding around a model — memory, tools, planning, orchestration, observability, evaluation, governance, and security — that turns a raw model into a reliable agentic system. - **Harness Engineering:** the discipline responsible for building reliable agentic systems for enterprise environments by designing and operating the harness. - **Model / LLM:** the underlying large language model that performs reasoning and generation; in the harness it is treated as a powerful but non-deterministic, manipulable component. - **Autonomy gradient:** the spectrum of control modes from fully autonomous to human-in-the-loop, assigned per action class by policy. - **Blast radius:** the maximum harm an action (or a compromised agent) can cause; a core quantity to bound. ### Tools and actions - **Tool:** a function or capability the agent can invoke to observe or affect the world (search, code execution, API call, database query). - **Tool call / function call:** a structured request from the model to invoke a tool with arguments. - **Effector:** a tool that produces a side effect in an external system (sends, writes, transfers). - **Idempotency:** the property that performing an operation multiple times has the same effect as performing it once; essential for safe retries and resumes. - **Side effect:** an externally visible change produced by a tool call. - **Allowlist:** an explicit set of permitted items (tools, egress destinations); the safe default for security-sensitive choices. ### Memory and context - **Context window:** the bounded span of tokens the model can attend to in a single inference. - **Memory:** the harness layer that persists and retrieves information across steps and sessions (short-term, long-term, episodic, semantic). - **RAG (Retrieval-Augmented Generation):** supplying the model with relevant retrieved content at inference time so its output is grounded in a corpus rather than parametric memory alone. - **Embedding:** a vector representation of text used for semantic similarity search in retrieval. - **Vector store:** a database optimized for nearest-neighbor search over embeddings. - **Grounding:** anchoring generated claims in retrieved, citable evidence. - **Context engineering:** the practice of deciding precisely what information enters the context window for each step. ### Planning - **Goal:** the desired end-state with explicit success criteria. - **Plan:** an ordered or partially-ordered set of tasks expected to achieve a goal. - **Decomposition:** breaking a goal into sub-tasks (see PAT-010, HRN-009). - **DAG (Directed Acyclic Graph):** a plan representation that captures task dependencies and exposes parallelism. - **Replanning:** revising a plan in response to failure, new information, or changed constraints. - **Termination criteria:** the budgets and conditions (steps, time, cost) that stop an agent safely. ### Orchestration - **Orchestration:** the harness layer that drives execution across agents and tools (see HRN-010). - **Topology:** the arrangement of agents — single, pipeline, supervisor/worker, or network. - **Supervisor (orchestrator) agent:** an agent that plans and delegates to worker agents (see PAT-002). - **Worker agent:** a specialized agent that executes a delegated sub-task (see PAT-005). - **Routing:** selecting the next agent, tool, or branch based on state. - **Handoff:** transfer of control and context from one agent to another. - **State machine:** an explicit graph of states and governed transitions controlling execution. - **Durable execution:** workflow semantics where progress is checkpointed and resumable across failures. - **Saga:** a sequence of operations with compensating actions to undo partial work on failure. ### Governance and security - **Governance:** the runtime harness layer enforcing policy, approvals, and guardrails (see HRN-008). - **Policy-as-code:** governance rules expressed in a declarative, versioned, testable format. - **PEP / PDP:** Policy Enforcement Point (intercepts actions) and Policy Decision Point (evaluates policy). - **Guardrail:** a runtime check on inputs or outputs that constrains agent behavior. - **Approval gate:** a control that suspends execution pending a human or higher-authority decision (see PAT-001). - **Least privilege:** granting each agent the minimum permissions required for its task. - **Agent identity:** a distinct, attributable, scoped, revocable principal for each agent. - **Prompt injection:** untrusted content that hijacks an agent's instructions or goals. - **Indirect prompt injection:** injection delivered via data the agent retrieves. - **Data exfiltration:** unauthorized egress of sensitive data through agent outputs or tool arguments. - **Lethal trifecta:** the dangerous combination of private-data access, untrusted-content exposure, and external communication ability. - **Sandbox:** an isolated, resource- and network-constrained environment for executing untrusted tools/code. - **DLP (Data Loss Prevention):** controls that detect and block sensitive data in outbound payloads. ### Observability and evaluation - **Observability:** the harness layer that makes agent behavior inspectable via traces, logs, and metrics (see HRN-006). - **Trace:** the end-to-end record of a single agent run. - **Span:** a single timed unit of work within a trace (one tool call, one model call), the building block of distributed tracing. - **Evaluation (eval):** the systematic measurement of agent quality, safety, and reliability (see HRN-007). - **LLM-as-judge:** using a model to score another model's outputs against a rubric. - **Groundedness:** the degree to which generated claims are supported by provided evidence. - **Regression suite:** a fixed set of evaluation cases run on every change to catch quality regressions. - **Eval-driven development:** building and changing agents against a measurable evaluation harness. ### Protocols and standards - **MCP (Model Context Protocol):** an open protocol for connecting models/agents to tools and data sources through a standardized interface. - **Tool schema:** the typed declaration of a tool's name, arguments, and description that the model uses to call it. - **llms.txt:** a proposed convention for a site-level Markdown file that surfaces a curated, agent-friendly index of content. - **JSON-LD:** structured data format used to make documents machine-readable in the discovery layer. - **NIST AI RMF / ISO 42001 / EU AI Act:** the principal AI risk, management-system, and regulatory frameworks an enterprise harness must satisfy (see HRN-014, GOV-001). ## FAQs **Q: Where do I cite a definition from?** A: Cite the term by name plus `HRN-013`. Where a term has a dedicated chapter, prefer citing that chapter for depth. **Q: A term I need is missing — what do I do?** A: Add it here in the appropriate group with a crisp one-sentence definition, and link the dedicated chapter if one exists. --- ## HRN-014 — Bibliography URL: https://santismm.com/en/handbook/bibliography | API: https://santismm.com/api/handbook/HRN-014 Category: Reference | Updated: 2026-06-21 Evidence: theoretical | Confidence: high | Source: industry_observation, paper | Status: Draft Related: HRN-001, GOV-001 # Bibliography ## Executive Summary This bibliography is the curated reference list underpinning the Harness Engineering handbook. It is organized by theme so a reader can go deep on any single layer. Entries are real, well-known works and standards. Where exact citation details (DOIs, page numbers) are not asserted here, the title and venue/source are given without fabricating identifiers; readers should confirm current versions of evolving standards. ## Definition The following references are grouped by theme. They are the primary sources the handbook draws on and the recommended starting points for further study. ### 1. Foundations of agents and reasoning - Yao, S. et al. **"ReAct: Synergizing Reasoning and Acting in Language Models."** ICLR. The reasoning-and-acting interleaving that underpins tool-using agents. - Wei, J. et al. **"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models."** NeurIPS. Foundational to step-by-step reasoning. - Schick, T. et al. **"Toolformer: Language Models Can Teach Themselves to Use Tools."** NeurIPS. - Shinn, N. et al. **"Reflexion: Language Agents with Verbal Reinforcement Learning."** NeurIPS. The reflection/self-critique loop (cf. PAT-003). - Wang, L. et al. **"A Survey on Large Language Model based Autonomous Agents."** A broad survey of the agent design space. - Wang, X. et al. **"Plan-and-Solve Prompting."** ACL. Plan-then-execute decomposition. ### 2. Orchestration, multi-agent, and durable execution - Anthropic. **"Building Effective Agents."** Engineering guidance on workflows vs. agents and single-agent-first design. - Anthropic. **"How we built our multi-agent research system."** Practical supervisor/worker orchestration writeup. - LangChain. **LangGraph documentation** — state-machine orchestration for agents. - Temporal / durable-execution engines. **Documentation on workflow durability and the Saga pattern.** - Microsoft / AutoGen. **"AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation."** Multi-agent conversation framework. - Hong, S. et al. **"MetaGPT: Meta Programming for Multi-Agent Collaborative Framework."** ### 3. Memory and retrieval (RAG) - Lewis, P. et al. **"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks."** NeurIPS. The canonical RAG paper. - Gao, Y. et al. **"Retrieval-Augmented Generation for Large Language Models: A Survey."** - Packer, C. et al. **"MemGPT: Towards LLMs as Operating Systems."** Memory hierarchy and paging for agents. - Asai, A. et al. **"Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection."** ### 4. Evaluation - Liang, P. et al. **"Holistic Evaluation of Language Models (HELM)."** Stanford CRFM. - Zheng, L. et al. **"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena."** The LLM-as-judge methodology and its biases. - Es, S. et al. **"RAGAS: Automated Evaluation of Retrieval Augmented Generation."** Groundedness and faithfulness metrics. - Liu, Y. et al. **"G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment."** - **SWE-bench** and **GAIA** — agentic capability benchmarks for software and general assistance tasks. ### 5. Security for agentic systems - OWASP. **"OWASP Top 10 for Large Language Model Applications."** Including LLM01 Prompt Injection and LLM06 Sensitive Information Disclosure. - Willison, S. **"Prompt injection"** and **"The lethal trifecta for AI agents"** (blog essays). The clearest articulation of the architectural injection/exfiltration problem. - Greshake, K. et al. **"Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection."** - MITRE. **ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems).** - NIST. **SP 800-53** (security and privacy controls; least privilege, identity) as adapted for AI systems. ### 6. Governance, risk, and regulatory standards - NIST. **AI Risk Management Framework (AI RMF 1.0)** and the **Generative AI Profile.** - ISO/IEC. **42001:2023 — Artificial intelligence — Management system.** - ISO/IEC. **23894:2023 — AI — Guidance on risk management.** - European Union. **Regulation (EU) 2024/1689, the EU AI Act** — risk-tiered obligations for AI systems. - OECD. **OECD AI Principles.** - US White House / OMB. **Executive and management guidance on trustworthy AI** (for context on public-sector expectations). - See also GOV-001 (Enterprise AI Governance Framework) and GOV-005 (Agent Governance Controls Checklist) in this corpus. ### 7. Protocols, interoperability, and the discovery layer - Anthropic. **Model Context Protocol (MCP) specification** — standardized model-to-tool/data interface. - **llms.txt** proposal — a site-level convention for agent-friendly content indexing. - **JSON-LD / schema.org** — structured data for machine discovery. - **OpenAPI Specification** — typed contracts for tools exposed as APIs. ### 8. Authorization and policy enforcement (adapted from systems engineering) - OASIS. **eXtensible Access Control Markup Language (XACML)** — the PEP/PDP authorization model. - **Open Policy Agent (OPA) / Rego** — policy-as-code engine. - Saltzer, J. & Schroeder, M. **"The Protection of Information in Computer Systems."** Origin of the least-privilege principle. ## FAQs **Q: Why are some entries missing DOIs or exact dates?** A: To avoid fabricating identifiers. Titles and venues/sources are given so the work is unambiguously locatable; confirm the current version, especially for evolving standards (EU AI Act, NIST AI RMF, MCP, OWASP). **Q: How does this relate to GOV-001?** A: GOV-001 operationalizes the governance and regulatory sources in sections 5–6 into an enterprise framework; this bibliography is the underlying reading list. **Q: Where should a newcomer start?** A: Section 1 (ReAct, Reflexion) for how agents work, section 2 (Building Effective Agents) for how to engineer them reliably, and sections 5–6 for security and governance. ================================================================= # KNOWLEDGE BASE ================================================================= ## What are Agent Memory Systems? URL: https://santismm.com/en/knowledge/agent-memory | API: https://santismm.com/api/knowledge/agent-memory Category: harness | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper **Definition.** Agent memory is the set of mechanisms an AI agent uses to store, organize and retrieve information beyond its immediate context window, spanning short-term working memory and long-term persistent memory. Agent memory is how an AI agent retains and recalls information beyond a single context window — across steps, sessions and tasks. It typically separates short-term working memory (the current context) from long-term memory (durable stores the agent reads from and writes to). Memory is what lets an agent carry state through a long task, remember a user over time, and avoid repeating work. It is a core layer of harness engineering. ### Key takeaways - Memory extends an agent beyond a single finite context window. - Short-term (working) vs long-term (persistent) memory serve different roles. - Long-term memory is often retrieved on demand, like RAG over the agent's own history. - Good memory prevents repeated work and lost state on long tasks. - What to write, keep and forget is a design decision, not automatic. ### Context A model's context window is finite, so anything an agent must remember beyond it needs an external store. Without memory, an agent forgets earlier steps, repeats actions and cannot personalize across sessions. Memory turns a stateless model into a system with continuity. The hard part is curation: deciding what is worth writing down, how to organize it, and how to retrieve only what is relevant now — closely tied to context engineering. ### Architecture Two layers: working memory (the current context window, holding the active task) and long-term memory (external stores — vector, key-value, document or graph — written during a task and retrieved later). Some designs add episodic (events), semantic (facts) and procedural (skills) memory. The agent reads relevant memories into context at each step and writes new ones as it learns. Retrieval, summarization and forgetting policies keep memory useful rather than overwhelming. ### Components - Working memory - Long-term store (vector/KV/graph) - Write policy - Retrieval policy - Summarization / consolidation - Forgetting / expiry ### Benefits - Continuity across steps, sessions and tasks. - Personalization that persists over time. - Avoids repeated work and lost context. - Enables long-horizon, stateful agents. ### Risks - Stale or wrong memories poison future answers. - Privacy and governance obligations on stored data. - Retrieval of irrelevant memories adds noise. - Unbounded growth without consolidation or expiry. ### Tools & technologies - Vector databases - Key-value / document stores - MemGPT-style memory managers - Agent frameworks (LangGraph, Agents SDKs) ### Examples - An assistant remembering a user's preferences across sessions. - A long-running agent summarizing progress so it never repeats a step. - A support agent recalling a customer's prior tickets when relevant. ### FAQs **How is agent memory different from RAG?** RAG retrieves from an external knowledge base; agent memory retrieves from the agent's own accumulated state. The retrieval machinery is similar, but the source and write path differ. **What is the difference between short- and long-term memory?** Short-term (working) memory is the live context window for the current task; long-term memory is a persistent store the agent reads from and writes to across tasks and sessions. **Why not keep everything in the context window?** Windows are finite and quality drops as they fill. Externalizing state to memory and retrieving only what is relevant keeps the agent focused and affordable. **What are the governance concerns?** Stored memory may contain personal or sensitive data, so it needs access control, retention limits and the ability to correct or delete entries. ### References - Packer et al. — MemGPT: Towards LLMs as Operating Systems (2023) — https://arxiv.org/abs/2310.08560 - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents Related: harness-engineering, context-engineering, ai-agent, enterprise-rag --- ## What is Agentic AI? URL: https://santismm.com/en/knowledge/agentic-ai | API: https://santismm.com/api/knowledge/agentic-ai Category: concept | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper **Definition.** Agentic AI is the class of AI systems that autonomously plan and execute multi-step tasks by combining a model with memory, tools and a control loop. Agentic AI refers to systems that pursue goals over multiple steps — planning, calling tools, acting on an environment and reacting to feedback — instead of producing a single response. It turns a language model from a text generator into an actor that can complete tasks. The shift it represents is from do-it-yourself software, where the human drives every step, to do-it-for-me software, where the system carries out the work and reports back. ### Key takeaways - An agent = model + tools + memory + a control loop that decides what to do next. - Autonomy is a spectrum, from a single tool call to long-horizon task execution. - Reliability comes mostly from the harness around the model, not raw model IQ. - Tool use (function calling) is what connects the model to real systems and data. - Evaluation must measure task completion (agency), not just answer quality (capability). ### Context For most of the LLM era, models were used as one-shot responders: a prompt in, an answer out. Agentic AI breaks that pattern by giving the model a loop — it can decide to call a tool, read the result, revise its plan and continue until the goal is met or a budget is exhausted. This is the dominant frontier of applied AI in the enterprise because it moves the value from answering questions to completing work: resolving a support ticket end to end, refactoring a codebase, running a research task, operating a workflow. ### Architecture A minimal agent has four parts: a reasoning model, a set of tools it can invoke, some form of memory or state, and an orchestration loop that turns model outputs into actions and feeds observations back in. Patterns range from simple (a model with tools and a stop condition) to complex (planner-executor splits, reflection, and multi-agent teams). Anthropic's guidance is to prefer the simplest pattern that works and add structure only when measurably needed. ### Components - Reasoning model - Tools / function calling - Memory & state - Orchestration loop - Guardrails - Observability ### Benefits - Completes multi-step work, not just single answers. - Adapts to feedback and recovers from intermediate errors. - Integrates with real systems through tools and APIs. - Scales repetitive knowledge work that was previously human-only. ### Risks - Compounding errors over long task horizons. - Unbounded cost and latency without budgets and stop conditions. - Security exposure from tool access and prompt injection. - Hard to evaluate and debug compared with single-shot prompts. ### Tools & technologies - LangGraph - OpenAI Agents SDK - Claude Agent SDK - Model Context Protocol (MCP) - Vertex AI Agent Engine ### Examples - A customer-service agent that reads a ticket, looks up the order, applies a refund and replies — all through tools. - A coding agent that edits files, runs tests and iterates until the suite passes. - A research agent that searches, reads sources, verifies claims and writes a cited summary. ### FAQs **What is the difference between an AI agent and agentic AI?** An AI agent is a concrete system; agentic AI is the broader paradigm of building software around such goal-directed, multi-step systems. **Do you need a more powerful model to be agentic?** Not necessarily. The same model can succeed or fail at a task depending almost entirely on the harness — the tools, memory, prompts and control loop around it. **Is RAG agentic?** Plain retrieval-augmented generation is a single step. It becomes agentic when the system decides when and what to retrieve as part of a multi-step loop. **What makes agents unreliable?** Long horizons compound small errors, tools fail, and context gets lost. Reliability comes from harness engineering: good tools, memory, guardrails and evaluation. **How do you measure an agent?** With agentic benchmarks and task-based evals that score end-to-end task completion in an environment, not just the quality of a single answer. ### References - Yao et al. — ReAct: Synergizing Reasoning and Acting in Language Models (2022) — https://arxiv.org/abs/2210.03629 - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - Model Context Protocol — Introduction — https://modelcontextprotocol.io Related: ai-agent, harness-engineering, multi-agent-architecture, agentic-evaluation --- ## What is Agentic AI Evaluation? URL: https://santismm.com/en/knowledge/agentic-evaluation | API: https://santismm.com/api/knowledge/agentic-evaluation Category: concept | Updated: 2026-06-24 | Version: 1.1 Evidence: production | Confidence: medium | Source: production_system, personal_experience, industry_observation, paper **Definition.** Agentic evaluation is the measurement of an AI agent's end-to-end task performance — success rate, reliability, cost and safety — on realistic, multi-step tasks in an environment. Agentic AI evaluation is the practice of measuring how well an agent completes multi-step, tool-using tasks in an environment — not just the quality of a single answer. As models saturate static knowledge benchmarks, evaluation is shifting from measuring capability (what a model knows) to measuring agency (what a system can actually get done). Good evals are the feedback loop that makes harness engineering possible. ### Key takeaways - Evaluate task completion (agency), not just answer quality (capability). - Agentic benchmarks test tools, environments and long horizons. - Static benchmarks saturate; agentic ones are the new frontier. - Evals are the feedback loop for improving the harness. - Measure success, reliability, cost, latency and safety together. ### Context Traditional benchmarks ask a model questions and score the answers. That measures capability, but it tells you little about whether a system can complete real work. Agentic evaluation instead places an agent in an environment with tools and a goal, and scores whether it actually achieves it. This shift matters because production value comes from task completion. An agent that answers well but fails to finish tasks is not useful. Evaluation is also what lets teams improve harnesses systematically rather than by anecdote. ### Architecture An agentic eval defines tasks, an environment (real or simulated) with tools, a success criterion, and metrics. The agent runs; its trajectory and outcome are scored automatically where possible, with human review for nuanced cases. Beyond a single success rate, mature evaluation tracks reliability across runs, cost and latency budgets, and safety (did the agent stay within authorization and avoid harmful actions). Traces from observability feed directly into eval design. ### Components - Task suite - Environment & tools - Success criteria - Metrics (success, cost, latency, safety) - Automated graders - Human review - Trajectory traces ### Benefits - Measures what actually matters: task completion. - Catches regressions before they reach users. - Turns harness improvement into a measurable loop. - Surfaces reliability, cost and safety, not just accuracy. ### Risks - Hard to build realistic environments and graders. - Overfitting to a benchmark instead of real performance. - Saturation: benchmarks lose discriminative power over time. - Automated grading can miss nuance; human review is costly. ### Tools & technologies - SWE-bench and other agentic benchmarks - LangSmith / Langfuse - OpenAI Evals - Custom task harnesses - LLM-as-judge graders ### Examples - Scoring a coding agent on whether its patch makes a real test suite pass. - Measuring a support agent's end-to-end ticket resolution rate. - Tracking reliability of a workflow agent across repeated runs. - In practice: evaluating an autonomous agent (OpenClaw) over 57 days straight from production traces — 161 sessions at 98.8% success (2 errors), with per-model reliability of 100% vs 95.7% across two model versions — a reliability baseline computed directly from operational traces rather than a curated benchmark set. ### FAQs **What is the difference between capability and agency?** Capability is what a model knows or can do in isolation; agency is what a full system can actually accomplish in an environment. Agentic evaluation measures the latter. **Why are static benchmarks no longer enough?** Top models saturate them, so they stop discriminating. They also do not test tool use, environments or long-horizon tasks, which is where real agent performance lives. **What is an agentic benchmark?** A test that scores an agent's ability to complete multi-step, tool-using tasks in an environment — for example resolving real software issues. **How do evals relate to harness engineering?** Evals are the measurement loop that makes harness engineering possible: you change the harness, measure the effect, and keep what demonstrably improves task performance. ### References - Jimenez et al. — SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) — https://arxiv.org/abs/2310.06770 - Santiago Santa María — The Stopwatch and the Exam — https://articles.santismm.com/the-stopwatch-and-the-exam/ Related: agentic-ai, harness-engineering, ai-agent, ai-governance --- ## What is an AI Agent? URL: https://santismm.com/en/knowledge/ai-agent | API: https://santismm.com/api/knowledge/ai-agent Category: concept | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper **Definition.** An AI agent is a system that pairs a reasoning model with tools, memory and an orchestration loop so it can plan and act over multiple steps to achieve a goal. An AI agent is a system that combines a model with tools, memory and a control loop to take actions toward a goal, rather than just answering a single prompt. It perceives a situation, decides what to do, acts through tools, observes the result and repeats until done. Autonomy ranges from a single tool call to long-horizon execution of complex tasks. ### Key takeaways - Agent = model + tools + memory + control loop. - Agents act; chatbots answer. - Tool use is the bridge to real systems and data. - More autonomy means more need for guardrails and evaluation. - Reliability is engineered through the harness, not assumed from the model. ### Context A plain LLM produces text. An agent uses that text to decide and act: it can search, call APIs, write files or trigger workflows, then react to what happens. This loop is what lets it complete tasks instead of merely describing them. Agents sit on a spectrum of autonomy. Low-autonomy agents make one or two tool calls under tight control; high-autonomy agents run long, branching tasks with little supervision — and need correspondingly stronger guardrails. ### Architecture The core loop is sense → decide → act → observe. The model decides the next action, an orchestrator executes it via a tool, the result is fed back as an observation, and the loop continues until a goal or stop condition is reached. Around this loop sit memory (to carry state), guardrails (to constrain behavior), and observability (to trace what happened). These are the parts of the harness that make an agent dependable. ### Components - Reasoning model - Tools - Memory - Orchestration loop - Guardrails - Observability ### Benefits - Executes tasks end to end. - Connects models to live systems. - Recovers from intermediate failures. - Automates multi-step knowledge work. ### Risks - Error compounding over long tasks. - Cost and latency without budgets. - Prompt injection and tool-misuse risks. - Harder to test and debug than single prompts. ### Tools & technologies - LangGraph - Claude Agent SDK - OpenAI Agents SDK - CrewAI - Model Context Protocol (MCP) ### Examples - A triage agent that classifies and routes incoming requests. - A coding agent that fixes a bug and verifies it with tests. - A data agent that queries a warehouse and assembles a report. ### FAQs **Is a chatbot an AI agent?** Not by default. A chatbot answers; an agent takes actions through tools across multiple steps to reach a goal. **What is the simplest useful agent?** A model with one or two well-described tools and a clear stop condition. Start simple and add structure only when measurement shows you need it. **What makes an agent reliable?** The harness: clean tool design, good memory, guardrails, observability and evaluation — not just a stronger model. **Single agent or multi-agent?** Prefer a single agent until a task clearly benefits from specialized, separable roles. Multi-agent adds coordination overhead. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - Yao et al. — ReAct (2022) — https://arxiv.org/abs/2210.03629 Related: agentic-ai, harness-engineering, model-context-protocol, multi-agent-architecture --- ## What is AI Governance? URL: https://santismm.com/en/knowledge/ai-governance | API: https://santismm.com/api/knowledge/ai-governance Category: governance | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: paper, industry_observation **Definition.** AI governance is the framework of policies, processes, roles and controls an organization uses to manage AI risk and ensure AI systems are responsible, compliant, secure and accountable across their lifecycle. AI governance is the set of policies, processes, roles and controls that ensure AI is built and used responsibly, legally and safely. It spans risk management, accountability, transparency, security and compliance across the AI lifecycle. In practice it operationalizes recognized frameworks — the EU AI Act, the NIST AI Risk Management Framework and ISO/IEC 42001 — into concrete controls an organization can implement, evidence and audit. ### Key takeaways - Governance turns AI principles into enforceable, auditable controls. - It spans the full lifecycle: data, build, deploy, monitor, retire. - Major references: EU AI Act, NIST AI RMF, ISO/IEC 42001. - Agentic AI raises the stakes: autonomy and tool access widen risk. - Good governance enables adoption; it is not just a brake. ### Context As AI moves into decisions that affect people and operations, organizations need a way to manage its risks systematically — not ad hoc. Governance provides that: clear ownership, documented risk assessment, controls, monitoring and the evidence to demonstrate compliance. Agentic systems sharpen the need. When software can act autonomously and call tools, the questions of authorization, accountability, auditability and human oversight become operational, not theoretical. ### Architecture A practical governance program has layers: policy and principles; an AI risk catalog and assessment process; controls (access, data handling, evaluation, human oversight, logging); monitoring and incident response; and an audit trail that maps controls to external frameworks. Frameworks complement each other. The NIST AI RMF organizes practice around Govern, Map, Measure and Manage. ISO/IEC 42001 defines an auditable AI management system. The EU AI Act sets legal obligations by risk tier. Mature programs map their controls to all three. ### Components - Policy & principles - AI risk catalog - Risk assessment process - Controls (access, data, oversight) - Evaluation & monitoring - Incident response - Audit trail & framework mapping ### Benefits - Manages legal, ethical and operational risk systematically. - Builds trust with customers, regulators and employees. - Provides evidence for audits and certifications. - Enables faster, safer adoption with clear guardrails. ### Risks - Bureaucracy that slows adoption if over-engineered. - Paper compliance that does not change real behavior. - Fragmented ownership across legal, security and product. - Falling behind fast-moving regulation and capabilities. ### Tools & technologies - NIST AI RMF - ISO/IEC 42001 - EU AI Act compliance mapping - Model & system documentation (model cards) - AI evaluation & monitoring platforms ### Examples - An AI risk catalog with assessments per use case before deployment. - Human-in-the-loop approval controls for high-impact agent actions. - Logging and audit trails mapped to NIST AI RMF functions. ### FAQs **What frameworks should we follow?** Commonly the NIST AI Risk Management Framework, ISO/IEC 42001 and, where in scope, the EU AI Act. Mature programs map a single control set to all three. **Is AI governance only about compliance?** No. Compliance is one part. Governance also covers risk, security, transparency, accountability and operational oversight that enable safe adoption. **How does governance apply to agents?** Autonomy and tool access add risk, so agentic systems need authorization controls, human oversight for high-impact actions, logging and evaluation. **Does governance slow innovation?** Done well, it accelerates it: clear guardrails let teams ship with confidence. Done poorly, it becomes bureaucracy. The goal is enforceable, lightweight controls. ### References - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework - European Union — AI Act (Regulation 2024/1689) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj - ISO/IEC 42001:2023 — AI management systems — https://www.iso.org/standard/81230.html Related: enterprise-rag, agentic-evaluation, agentic-ai, harness-engineering --- ## What is AI Agent Observability? URL: https://santismm.com/en/knowledge/ai-observability | API: https://santismm.com/api/knowledge/ai-observability Category: harness | Updated: 2026-06-24 | Version: 1.1 Evidence: production | Confidence: medium | Source: production_system, personal_experience, industry_observation **Definition.** AI observability is the practice of capturing traces, metrics and logs of an AI system's behavior — every prompt, tool call, retrieval, output, token, latency and cost — so its decisions can be understood, debugged and improved. AI observability is the practice of instrumenting AI systems — especially agents — so you can see what they did and why. It captures traces of each step: prompts, tool calls, retrieved context, model outputs, tokens, latency and cost. Because agents are non-deterministic and multi-step, observability is what makes failures diagnosable and improvement systematic. It is the layer that feeds evaluation and closes the harness-engineering loop. ### Key takeaways - Observability makes non-deterministic agents debuggable. - Traces record each step: prompts, tools, context, outputs, cost. - It feeds evaluation — you improve what you can see and measure. - Track quality, latency, cost and safety together. - Emerging standards (OpenTelemetry GenAI) make traces portable. ### Context Traditional software is deterministic and easy to log. Agents are not: the same input can take different paths, call different tools and produce different outputs. Without tracing, a failure is a black box. Observability opens that box. By recording the full trajectory of a run, teams can see where an agent went wrong, why a tool failed, where cost ballooned — and feed those findings into evals and harness changes. ### Architecture Instrumentation captures spans for each step — model call, tool call, retrieval — with inputs, outputs, tokens, latency and errors, linked into a trace for the whole run. Metrics aggregate quality, cost, latency and failure rates over time. OpenTelemetry's GenAI semantic conventions standardize how these traces are structured, so they can flow into general observability backends rather than proprietary silos. Traces also become the raw material for evaluation datasets. ### Components - Tracing (spans per step) - Metrics (quality, cost, latency) - Logs - Token & cost accounting - Error tracking - Trace-to-eval pipeline ### Benefits - Turns opaque agent runs into diagnosable traces. - Surfaces cost, latency and failure hotspots. - Feeds evaluation and continuous improvement. - Supports incident response and governance audits. ### Risks - Traces may capture sensitive data needing redaction. - Instrumentation overhead and storage cost at scale. - Volume without good queries hides the signal. - Privacy and retention obligations on logged prompts. ### Tools & technologies - OpenTelemetry (GenAI conventions) - LangSmith - Langfuse - Arize / Phoenix - Standard APM backends ### Examples - Tracing a failed agent run to the exact tool call that errored. - Tracking per-task token cost to find an expensive prompt. - Turning production traces into an evaluation dataset. - In practice: a 57-day single-operator autonomous deployment (OpenClaw) traced 161 sessions / 2,776 turns from local trajectory traces — 98.8% session success, p50 8.1s and p95 87.6s per turn, ~70% of tokens served from cache, and a blended $15.21 per 1M tokens (cost partially priced, so a floor). Every figure is derived from the agent's own traces. ### FAQs **Why do agents need observability more than chatbots?** Agents are multi-step and non-deterministic, so a single answer hides many internal decisions. Without traces of those steps, failures cannot be diagnosed. **How does observability relate to evaluation?** Observability captures what happened; evaluation judges whether it was good. Traces become the data evals run on, closing the improvement loop. **Is there a standard for AI traces?** OpenTelemetry's generative-AI semantic conventions are emerging as a portable standard, letting AI traces flow into mainstream observability tooling. **What should you measure?** Quality (task success), cost (tokens), latency, and safety together — a fast, cheap agent that fails the task is not a good agent. ### References - OpenTelemetry — Semantic conventions for generative AI — https://opentelemetry.io/docs/specs/semconv/gen-ai/ - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents Related: harness-engineering, agentic-evaluation, ai-agent, ai-governance --- ## What is Context Engineering? URL: https://santismm.com/en/knowledge/context-engineering | API: https://santismm.com/api/knowledge/context-engineering Category: harness | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper **Definition.** Context engineering is the practice of curating, compressing and sequencing the information placed in a model's context window so it has the most relevant signal — and the least noise — at each step. Context engineering is the discipline of deciding what information enters a model's limited context window at each step — and what stays out. As agents run over many steps, naively stuffing everything into context degrades quality and cost. Context engineering curates the right instructions, retrieved knowledge, tool results and memory so the model has exactly what it needs, when it needs it. It is a core part of harness engineering. ### Key takeaways - Context is a scarce resource; what you leave out matters as much as what you include. - More context is not better — irrelevant tokens degrade quality and raise cost. - Techniques: retrieval, summarization, compaction, and structured memory. - It generalizes prompt engineering from one prompt to a whole agent run. - It is a core layer of the harness around a model. ### Context Every model has a finite context window, and quality degrades when it is filled with low-signal content. In single-turn use this is manageable, but agents accumulate history, tool outputs and retrieved documents across many steps, quickly overwhelming the window. Context engineering treats the window as a budget to be managed deliberately: keep the durable instructions, retrieve only what is relevant now, summarize or compact the rest, and store long-term state outside the window in memory. ### Architecture Core moves: select (retrieve only relevant passages), compress (summarize prior steps), compact (drop or fold stale turns), and externalize (push long-term state to a memory store, pulling it back on demand). In an agent loop, context is reassembled each step from layered sources: stable system instructions, task state, relevant retrieved knowledge, recent tool results and selected long-term memories — ordered so the most important signal is most salient. ### Components - System instructions - Task state - Retrieved knowledge - Tool results - Long-term memory - Summaries / compaction ### Benefits - Keeps quality high as tasks grow long. - Controls token cost and latency. - Reduces distraction and hallucination from noise. - Enables long-horizon agents within finite context. ### Risks - Over-aggressive compression can drop needed information. - Poor retrieval injects irrelevant or wrong context. - Complexity in deciding what to keep each step. - Bugs here surface as subtle quality regressions. ### Tools & technologies - Retrieval / RAG pipelines - Summarization models - Memory stores - Context-management frameworks (e.g. LangGraph) ### Examples - Summarizing earlier agent steps so the window stays focused on the current subtask. - Retrieving only the policy section relevant to a question instead of the whole manual. - Storing a user's preferences in memory and recalling them only when relevant. ### FAQs **How is context engineering different from prompt engineering?** Prompt engineering crafts a single instruction. Context engineering manages the full set of information in the window across an entire agent run — retrieval, memory, tool results and compression included. **Why not just use a bigger context window?** Larger windows help but do not eliminate the problem: quality and cost degrade as windows fill with low-signal tokens. Curation still wins. **How does it relate to RAG and memory?** RAG and memory are sources of context; context engineering decides what from them actually enters the window, when, and in what form. **Is it part of harness engineering?** Yes. Context management is one of the core layers of the harness that turns model capability into reliable agent behavior. ### References - Anthropic — Effective context engineering for AI agents (2025) — https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents Related: harness-engineering, prompt-engineering, agent-memory, enterprise-rag --- ## What are Embeddings & Vector Search? URL: https://santismm.com/en/knowledge/embeddings | API: https://santismm.com/api/knowledge/embeddings Category: concept | Updated: 2026-06-21 | Version: 1.0 Evidence: benchmark | Confidence: high | Source: benchmark, paper **Definition.** An embedding is a dense numeric vector that encodes the meaning of a piece of data, positioned so that similar items are close in vector space; vector search retrieves the nearest embeddings to a query. An embedding is a numeric vector that represents the meaning of text (or images, audio, code) so that semantically similar items sit close together in vector space. Vector search finds the nearest embeddings to a query, enabling search by meaning rather than keywords. Embeddings are the backbone of retrieval-augmented generation, semantic search, clustering and recommendation. ### Key takeaways - Embeddings turn meaning into vectors; similar things sit close together. - Vector search retrieves by semantic similarity, not exact keywords. - They power RAG, semantic search, clustering and recommendation. - Hybrid search (vector + keyword) usually beats either alone. - Chunking and the embedding model choice drive retrieval quality. ### Context Computers compare numbers, not meaning. Embeddings bridge that gap: an embedding model maps text to a vector such that 'cancel my subscription' and 'how do I unsubscribe' land near each other, even with no shared words. This is what makes semantic retrieval possible. Instead of matching keywords, a system embeds the query and finds the closest stored vectors — the foundation of how RAG and modern search retrieve relevant content. ### Architecture Indexing: content is split into chunks, each passed through an embedding model to produce a vector, then stored in a vector index. Querying: the query is embedded and the index returns the nearest vectors by a similarity metric (e.g. cosine). Production systems add a re-ranker to refine the top results, combine vector search with keyword search (hybrid), and filter by metadata and permissions. Quality depends heavily on chunking and the embedding model. ### Components - Embedding model - Chunking - Vector index / database - Similarity metric (cosine) - Re-ranker - Hybrid (keyword) search ### Benefits - Search by meaning, robust to wording. - Cross-lingual and multimodal matching. - The backbone of RAG and semantic search. - Cheap to query at scale once indexed. ### Risks - Poor chunking degrades every downstream result. - Embedding model mismatch hurts relevance. - Vectors can leak sensitive info; secure the store. - Pure vector search can miss exact-match needs (use hybrid). ### Tools & technologies - Embedding models (OpenAI, Cohere, open-source) - Vector databases (pgvector, Pinecone, Vertex AI Vector Search) - Re-rankers - Hybrid search engines ### Examples - Semantic search over a help center that matches intent, not keywords. - Retrieving relevant passages to ground a RAG answer. - Clustering support tickets by topic using their embeddings. ### FAQs **How are embeddings different from keywords?** Keyword search matches exact words; embeddings match meaning, so paraphrases and synonyms still retrieve the right content. **What is vector search?** Finding the stored embeddings nearest to a query embedding by a similarity metric like cosine distance — search by semantic closeness. **Why combine vector and keyword search?** Vectors excel at meaning but can miss exact terms (codes, names). Hybrid search blends both for the best recall and precision. **Do embeddings power RAG?** Yes. RAG embeds documents and queries, retrieves the nearest chunks by vector search, and grounds the model's answer in them. ### References - Reimers & Gurevych — Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019) — https://arxiv.org/abs/1908.10084 - Mikolov et al. — Efficient Estimation of Word Representations (word2vec, 2013) — https://arxiv.org/abs/1301.3781 Related: enterprise-rag, foundation-models, fine-tuning, agentic-ai --- ## What is Enterprise RAG? URL: https://santismm.com/en/knowledge/enterprise-rag | API: https://santismm.com/api/knowledge/enterprise-rag Category: pattern | Updated: 2026-06-21 | Version: 1.0 Evidence: benchmark | Confidence: high | Source: benchmark, paper, industry_observation **Definition.** Enterprise RAG is a pattern that retrieves relevant passages from an organization's governed knowledge sources and supplies them to a model as context, so answers are grounded, current and citable. Enterprise RAG (retrieval-augmented generation) is the pattern of grounding a model's answers in an organization's own documents, retrieved at query time, instead of relying on the model's parametric memory. It lets a company use private, current and governed knowledge — policies, manuals, tickets, contracts — without retraining a model, while keeping access control, citations and auditability that enterprises require. ### Key takeaways - RAG grounds answers in retrieved documents, reducing hallucination. - It uses private and fresh knowledge without retraining. - Retrieval quality (chunking + embeddings) drives answer quality. - Enterprise-grade RAG adds access control, citations and audit. - Becomes agentic when the system decides when and what to retrieve. ### Context A base model only knows what it learned during training. Enterprise knowledge is private, changing and access-controlled. RAG bridges that gap by fetching the right passages at query time and grounding the answer in them. The enterprise difference is governance: who is allowed to see which documents, where the answer's sources came from, and whether the whole interaction can be audited. RAG that ignores these is a prototype, not a production system. ### Architecture Ingestion: documents are parsed, split into self-contained chunks, embedded and stored in a vector index (often alongside keyword search). Retrieval: a query is embedded, the nearest chunks are fetched, optionally re-ranked and filtered by permissions. Generation: the model answers using those chunks and cites them. Quality hinges on the unglamorous parts: clean parsing, sensible chunking, hybrid (vector + keyword) retrieval, re-ranking, and permission filtering. Well-structured source content makes every one of these steps easier. ### Components - Ingestion & chunking - Embeddings - Vector / hybrid index - Retriever & re-ranker - Permission filter - Generator (LLM) - Citation layer ### Benefits - Grounded, citable, up-to-date answers. - Uses private knowledge without retraining. - Respects access control and auditability. - Cheaper and faster to update than fine-tuning. ### Risks - Poor chunking or retrieval yields wrong or irrelevant context. - Stale or unpermissioned data leaks into answers. - Citations can be plausible but unsupported if not verified. - Retrieval latency and cost at scale. ### Tools & technologies - Vector databases (e.g. pgvector, Pinecone, Vertex AI Vector Search) - Embedding models - Re-rankers - Hybrid search engines - MCP resource servers ### Examples - An internal assistant answering HR policy questions with cited passages. - A support agent retrieving product docs to resolve tickets. - A legal assistant surfacing relevant clauses with source links. ### FAQs **Is RAG better than fine-tuning?** They solve different problems. RAG injects fresh, governed knowledge at query time; fine-tuning adapts behavior or style. They are often combined. **Why does chunking matter so much?** Retrieval works on chunks. Self-contained, well-structured chunks retrieve cleanly; fragmented ones return noise. Chunk quality largely sets RAG quality. **What makes RAG enterprise-grade?** Access control on retrieval, source citations, auditability, freshness, and evaluation — not just a vector store plus a model. **When does RAG become agentic?** When retrieval is one step in a multi-step loop where the system decides whether, when and what to retrieve, rather than always retrieving once. ### References - Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) — https://arxiv.org/abs/2005.11401 - Model Context Protocol — Resources — https://modelcontextprotocol.io Related: agentic-ai, model-context-protocol, ai-governance, ai-agent --- ## What is Fine-tuning? URL: https://santismm.com/en/knowledge/fine-tuning | API: https://santismm.com/api/knowledge/fine-tuning Category: concept | Updated: 2026-06-21 | Version: 1.0 Evidence: benchmark | Confidence: high | Source: benchmark, paper **Definition.** Fine-tuning is the process of further training a pretrained model on a focused dataset to adapt its weights toward a specific behavior, style, format or domain. Fine-tuning continues training a pretrained model on a smaller, targeted dataset to specialize its behavior, style or domain knowledge. It is far cheaper than pretraining and changes the model's weights — unlike prompting or retrieval, which leave the model unchanged. Use it to lock in a consistent format, tone or skill; use retrieval instead when you need fresh or private facts. ### Key takeaways - Fine-tuning updates model weights; prompting and RAG do not. - Best for consistent behavior, style or format — not for fresh facts. - Parameter-efficient methods (LoRA) make it cheap and practical. - RLHF is a form of fine-tuning using human preferences. - Default to prompting and retrieval first; fine-tune when they plateau. ### Context A pretrained foundation model is a generalist. Fine-tuning narrows it: shown enough examples of the target behavior, the model internalizes it, so you no longer need to specify it in every prompt. It is one of three adaptation levers, alongside prompting and retrieval. The art is choosing the right one: fine-tune for how the model should behave, retrieve for what it should know. ### Architecture Full fine-tuning updates all weights — powerful but expensive. Parameter-efficient fine-tuning (PEFT), notably LoRA, trains small adapter weights while freezing the base, capturing most of the benefit at a fraction of the cost. Instruction tuning and RLHF are specialized fine-tuning stages that turn a raw base model into a helpful, aligned assistant. Quality of the dataset matters far more than its size. ### Components - Pretrained base model - Curated training dataset - Training objective - PEFT / LoRA adapters - Evaluation set ### Benefits - Bakes in consistent behavior, style or format. - Reduces prompt length and per-call cost. - Can teach narrow skills a base model lacks. - PEFT makes it affordable and fast. ### Risks - Does not add fresh or private facts — use retrieval for that. - Risk of catastrophic forgetting or overfitting. - Needs a quality, well-labeled dataset and an eval set. - Couples you to a model version; migration costs on upgrades. ### Tools & technologies - LoRA / PEFT libraries - Provider fine-tuning APIs - RLHF / preference-tuning pipelines - Evaluation suites ### Examples - Fine-tuning a model to always output a strict company JSON format. - Teaching a consistent brand voice for generated copy. - Adapting a model to a specialized domain's terminology. ### FAQs **Fine-tuning or RAG?** Fine-tune to change how the model behaves (style, format, skill); use retrieval (RAG) to give it fresh or private knowledge. They are complementary, not competing. **Is fine-tuning expensive?** Full fine-tuning can be, but parameter-efficient methods like LoRA train tiny adapters and make it cheap and fast for most use cases. **What is RLHF?** Reinforcement learning from human feedback is a fine-tuning stage that uses human preference judgments to make a model more helpful, harmless and honest. **When should I fine-tune?** After prompting and retrieval plateau. If you can solve it with a better prompt or relevant context, do that first — it is cheaper and more flexible. ### References - Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models (2021) — https://arxiv.org/abs/2106.09685 - Ouyang et al. — Training language models to follow instructions with human feedback (InstructGPT, 2022) — https://arxiv.org/abs/2203.02155 Related: foundation-models, enterprise-rag, prompt-engineering, embeddings --- ## What are Foundation Models? URL: https://santismm.com/en/knowledge/foundation-models | API: https://santismm.com/api/knowledge/foundation-models Category: concept | Updated: 2026-06-21 | Version: 1.0 Evidence: benchmark | Confidence: high | Source: benchmark, paper **Definition.** A foundation model is a large, general-purpose model pretrained on broad data that serves as a base which can be adapted — via prompting, retrieval or fine-tuning — to many downstream tasks. A foundation model is a large model pretrained on broad data at scale that can be adapted to a wide range of downstream tasks. Large language models (LLMs) and large multimodal models are the canonical examples. The term, coined at Stanford in 2021, captures a shift: instead of training a bespoke model per task, organizations build on a shared, general-purpose base — then specialize it through prompting, retrieval or fine-tuning. ### Key takeaways - Foundation models are general bases adapted to many tasks. - LLMs and multimodal models are the leading examples. - Most are built on the transformer architecture. - Capabilities emerge with scale of data, parameters and compute. - You adapt them by prompting, retrieval (RAG) or fine-tuning — rarely by training from scratch. ### Context Before foundation models, teams trained narrow models for each task. The foundation-model paradigm flips this: one large model is pretrained once on broad data, then reused everywhere. That reuse is why a handful of models now underpin most AI products. It also concentrates capability — and risk. Because so much is built on a few bases, their biases, failures and security properties propagate downstream, which is part of why governance and evaluation matter. ### Architecture Pretraining: a model with millions to trillions of parameters learns general patterns from massive datasets, typically with self-supervised objectives like next-token prediction. The transformer's attention mechanism makes this scalable. Adaptation: the same base is specialized for use — zero/few-shot prompting, retrieval-augmented generation for fresh or private knowledge, or fine-tuning for behavior and domain. Agents wrap the model in tools and a harness. ### Components - Transformer architecture - Pretraining data & objective - Parameters (weights) - Tokenizer - Adaptation layer (prompt / RAG / fine-tune) ### Benefits - One base reused across many tasks. - Strong general capability out of the box. - Rapid adaptation without training from scratch. - Multimodal variants span text, image, audio and more. ### Risks - Concentrated risk: flaws propagate to everything built on them. - Costly to pretrain; few organizations can. - Inherit biases and gaps from training data. - Knowledge is frozen at training time without retrieval. ### Tools & technologies - Frontier LLMs (Claude, GPT, Gemini) - Open-weight models (Llama, Mistral) - Multimodal models - Model hosting / inference platforms ### Examples - Using one LLM for summarization, classification and drafting across an org. - Adapting a base model to a domain with retrieval instead of retraining. - Building an agent on a frontier model plus tools and memory. ### FAQs **Is a foundation model the same as an LLM?** An LLM is the most common type of foundation model, specialized to language. Foundation models also include multimodal and other general-purpose models. **Why are they called 'foundation' models?** Because they serve as a shared base that many applications are built on, rather than a model trained for a single task. **Do I need to train one?** Almost never. Pretraining is extremely costly; nearly all value comes from adapting an existing base via prompting, retrieval or fine-tuning. **How do agents relate to foundation models?** An agent uses a foundation model as its reasoning core, wrapped in tools, memory and a control loop — the harness — to take actions. ### References - Bommasani et al. — On the Opportunities and Risks of Foundation Models (2021) — https://arxiv.org/abs/2108.07258 - Vaswani et al. — Attention Is All You Need (2017) — https://arxiv.org/abs/1706.03762 Related: fine-tuning, embeddings, agentic-ai, harness-engineering --- ## What are AI Guardrails? URL: https://santismm.com/en/knowledge/guardrails | API: https://santismm.com/api/knowledge/guardrails Category: governance | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper **Definition.** AI guardrails are runtime safeguards that validate, filter or constrain a model's inputs, outputs and actions to keep its behavior safe, compliant and within defined policy. Guardrails are runtime controls that constrain what goes into and comes out of an AI system, keeping its behavior safe, on-policy and compliant. They check and filter inputs and outputs, validate tool actions, block disallowed content and enforce limits — sitting around the model as a safety layer. Guardrails are a primary, operational control in AI governance and a key defense against misuse and prompt injection. ### Key takeaways - Guardrails act at runtime on inputs, outputs and actions. - They enforce safety, policy and compliance, not model quality. - Types: input filtering, output validation, action allow-lists, limits. - A core defense against misuse and prompt injection. - Guardrails complement — they do not replace — evaluation and oversight. ### Context A model alone has no enforceable boundaries; it will attempt whatever the prompt elicits. Guardrails add those boundaries operationally: deterministic or model-based checks that sit between the user, the model and the systems it can touch. They are how governance policies become live controls. A policy that says 'never expose PII' or 'never execute payments without approval' is realized as a guardrail that actually checks and blocks at runtime. ### Architecture Input guardrails screen prompts (e.g. for injection, policy violations, PII). Output guardrails validate responses (format, safety, factual constraints, PII redaction). Action guardrails gate tool calls with permissions and allow-lists. Rate, scope and budget limits cap blast radius. Guardrails can be deterministic (rules, schemas, regex, allow-lists) or model-based (a classifier or LLM judge). They pair with observability to log violations and with human-in-the-loop approval for high-impact actions. ### Components - Input filtering - Output validation / redaction - Action allow-lists & permissions - Rate & scope limits - Policy classifiers - Violation logging ### Benefits - Enforces safety and policy at runtime, not just in guidance. - Reduces misuse, unsafe output and injection impact. - Operationalizes governance and compliance requirements. - Bounds the blast radius of agent actions. ### Risks - Over-blocking harms usefulness (false positives). - Under-blocking creates a false sense of safety. - Model-based guardrails add latency and cost. - They are not a complete defense; combine with oversight and evals. ### Tools & technologies - Guardrail frameworks (e.g. NeMo Guardrails, Guardrails AI) - Content moderation / safety classifiers - Schema & input validation - Permission & policy engines - Observability for violations ### Examples - Redacting personal data from a model's output before it is shown. - Blocking a tool call that falls outside an allow-list of safe actions. - Rejecting responses that do not match a required JSON schema. ### FAQs **Are guardrails the same as alignment?** No. Alignment shapes the model's intrinsic behavior during training; guardrails are external runtime controls around the deployed system. They are complementary. **Do guardrails stop prompt injection?** They reduce its impact — input screening and action allow-lists help — but no guardrail fully prevents injection. Use layered defenses plus human approval for sensitive actions. **Deterministic or model-based guardrails?** Both. Deterministic checks (schemas, allow-lists) are cheap and reliable for clear rules; model-based checks handle nuanced content at the cost of latency. **How do guardrails fit AI governance?** They are the operational layer: the runtime controls that turn governance policies into enforced behavior, evidenced through logging and audit. ### References - OWASP — Top 10 for LLM Applications — https://genai.owasp.org/llm-top-10/ - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework Related: prompt-injection, ai-governance, human-in-the-loop, ai-observability --- ## What is Harness Engineering? URL: https://santismm.com/en/knowledge/harness-engineering | API: https://santismm.com/api/knowledge/harness-engineering Category: harness | Updated: 2026-06-21 | Version: 1.0 Evidence: theoretical | Confidence: medium | Source: personal_experience, industry_observation **Definition.** Harness engineering is the practice of designing, building and optimizing the scaffolding (tools, memory, prompts, environment and control loop) that turns a model's raw capability into reliable, goal-directed action. Harness engineering is the discipline of designing and optimizing the scaffolding around an AI model — the prompts, tools, memory, environment, control loop and guardrails — so the model performs reliably on real tasks. Its core premise: as base models converge in raw capability, competitive advantage shifts from the model itself to the harness built around it. The same model can pass or fail a task depending almost entirely on its harness. ### Key takeaways - The harness is everything around the model that converts capability into action. - As frontier models converge, the harness becomes the main lever of differentiation. - Tool design, context management and memory often matter more than model choice. - Harnesses must be observable and evaluated — you cannot improve what you cannot measure. - Harness engineering is to agents what platform engineering is to cloud applications. ### Context Benchmarks long measured a model's capability in isolation. But in production, a model never acts alone: it acts through a harness. Give a strong model a poor harness and it fails; give a modest model an excellent harness and it succeeds. That gap is where harness engineering lives. The term names a shift in where engineering effort and competitive advantage sit. When everyone can call a comparable frontier model, the durable advantage is the system around it: the quality of the tools, the memory, the context strategy, the evaluation loop and the guardrails. ### Architecture A harness has recurring layers: the prompt/instruction layer; the tool layer (what the model can do and how cleanly those tools are described); the memory layer (short-term context plus long-term stores); the environment (the systems the agent acts on); the control loop (how outputs become actions and observations return); and the cross-cutting layers of guardrails, observability and evaluation. Good harness engineering treats each layer as a design surface. Tools are written for a model to use, not just for a developer to read. Context is curated rather than dumped. Memory is structured. Every run is traced so failures can be diagnosed and fed back into evals. ### Components - Instruction / prompt layer - Tooling - Memory systems - Environment - Control loop / orchestration - Guardrails - Observability - Evaluation ### Benefits - Turns the same model into a far more reliable system. - A durable advantage that survives model upgrades and swaps. - Makes failures diagnosable through observability and evals. - Lets teams improve agents systematically, not by prompt luck. ### Risks - Complexity: more moving parts to build, secure and maintain. - Over-engineering harnesses that simpler patterns would solve. - Tight coupling to a model's quirks can create migration cost. - Without evaluation, harness changes are guesswork. ### Tools & technologies - LangGraph - Claude Agent SDK - OpenAI Agents SDK - Model Context Protocol (MCP) - LangSmith / Langfuse (observability) ### Examples - Rewriting a vague tool description so the model calls it correctly, lifting task success without touching the model. - Adding a memory store so an agent stops repeating work across a long task. - Introducing an evaluation harness that catches a regression before it ships. ### FAQs **Why does harness engineering matter now?** Because frontier models are converging. When raw capability is broadly available, the differentiator becomes the harness — the engineered system that turns that capability into dependable work. **Is harness engineering the same as prompt engineering?** No. Prompt engineering is one layer of the harness. Harness engineering also covers tools, memory, environment, the control loop, guardrails, observability and evaluation. **How is it different from agentic harness engineering?** Agentic harness engineering applies the same discipline specifically to autonomous, multi-step agents and their long-horizon needs (memory, tools, feedback loops). **What skills does it require?** Software and platform engineering, evaluation/measurement, systems design, security, and a working understanding of how models behave. **How do you know a harness is good?** By measuring it. A good harness is observable and evaluated against task-based benchmarks, so improvements are demonstrated rather than assumed. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - Yao et al. — ReAct (2022) — https://arxiv.org/abs/2210.03629 - Santiago Santa María — The Stopwatch and the Exam — https://articles.santismm.com/the-stopwatch-and-the-exam/ Related: agentic-ai, ai-agent, context-engineering, agent-memory, ai-observability, multi-agent-architecture, agentic-evaluation --- ## What is the Human-in-the-Loop Pattern? URL: https://santismm.com/en/knowledge/human-in-the-loop | API: https://santismm.com/api/knowledge/human-in-the-loop Category: pattern | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper **Definition.** Human-in-the-loop is a pattern in which a human reviews, approves, edits or rejects an AI system's proposed output or action before it is executed, inserting human judgment at defined decision points. Human-in-the-loop (HITL) is a design pattern where a person reviews, approves or corrects an AI system's output before it takes effect — especially for high-impact actions. Instead of full autonomy, the agent proposes and a human disposes. It is a primary control for managing risk in agentic systems and a recurring requirement in AI governance frameworks like the EU AI Act and NIST AI RMF. ### Key takeaways - The agent proposes; a human approves, edits or rejects. - Apply it to high-impact, irreversible or sensitive actions. - It trades some autonomy and speed for control and trust. - Governance frameworks often require human oversight by design. - Approvals and overrides should be logged for auditability. ### Context Full autonomy is risky when actions are costly, irreversible or regulated. HITL inserts a checkpoint: the AI does the work and a human makes the final call, capturing most of the efficiency while keeping accountability with a person. It is also a trust-building and adoption strategy. Teams often start with tight human review, then widen autonomy as evaluation shows the system is reliable for a given task. ### Architecture Patterns: human-in-the-loop (a person approves each high-impact action), human-on-the-loop (a person monitors and can intervene), and human-over-the-loop (periodic review and policy setting). The right level depends on the action's risk. Implementation needs an approval interface, clear context for the reviewer, the ability to edit or reject, fallback behavior on timeout, and logging of every decision for audit. ### Components - Decision checkpoints - Approval interface - Reviewer context - Edit / override - Escalation & fallback - Audit log ### Benefits - Catches errors before they cause harm. - Keeps accountability with a human. - Supports compliance and governance requirements. - Builds trust and enables gradual autonomy. ### Risks - Adds latency and limits throughput. - Rubber-stamping: reviewers approve without real scrutiny. - Alert fatigue degrades the quality of oversight. - Over-applying it negates the value of automation. ### Tools & technologies - Approval / workflow systems - Agent frameworks with interrupt steps (e.g. LangGraph) - Audit logging - Case-management UIs ### Examples - An agent drafting a refund that a human approves before it is issued. - A content agent whose output a person reviews before publishing. - An ops agent that pauses for sign-off before a production change. ### FAQs **When should you use human-in-the-loop?** For actions that are high-impact, irreversible, sensitive or regulated, where the cost of an error outweighs the latency of review. **What is the difference between in-the-loop and on-the-loop?** In-the-loop means a human approves each action before it executes; on-the-loop means a human monitors and can intervene, but the system acts on its own by default. **Does HITL conflict with autonomy?** It bounds autonomy deliberately. Many systems start with heavy review and widen autonomy as evaluation demonstrates reliability for a task. **Is it required by regulation?** Human oversight is a recurring requirement — for example the EU AI Act mandates effective human oversight for high-risk AI systems. ### References - European Union — AI Act, Article 14 (Human oversight) — https://artificialintelligenceact.eu/article/14/ - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework Related: ai-governance, agentic-ai, ai-agent, agentic-evaluation --- ## What is the Model Context Protocol (MCP)? URL: https://santismm.com/en/knowledge/model-context-protocol | API: https://santismm.com/api/knowledge/model-context-protocol Category: concept | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation **Definition.** MCP is an open protocol that standardizes how AI applications provide tools, resources and context to models through a common client–server interface. The Model Context Protocol (MCP) is an open standard for connecting AI models and agents to external tools, data sources and systems through a single, uniform interface. Introduced by Anthropic in late 2024, it standardizes how an application exposes context and capabilities to a model — acting like a universal adapter so any compliant client can talk to any compliant server. ### Key takeaways - MCP standardizes model-to-tool/data connections, like a universal adapter. - It uses a client–server model: hosts run clients; integrations are servers. - Servers expose tools, resources and prompts in a uniform shape. - It reduces N×M custom integrations to reusable, shareable connectors. - It is open and vendor-neutral, with growing multi-vendor adoption. ### Context Before MCP, every agent-to-system integration was bespoke: each tool wired by hand to each application. MCP replaces that with a shared protocol, so a connector written once can be reused across any MCP-aware client. This matters for enterprises because integration — not model quality — is often the real bottleneck in shipping agents. A common protocol turns connectors into a reusable ecosystem rather than one-off glue code. ### Architecture MCP defines three roles: a host application, an MCP client inside it, and one or more MCP servers. The client connects to servers over a transport, and servers expose capabilities the model can use. Servers offer three primitives: tools (actions the model can invoke), resources (data the model can read), and prompts (reusable templates). The model, via the host, discovers and uses these in a standard way. ### Components - Host application - MCP client - MCP server - Tools - Resources - Prompts - Transport ### Benefits - Eliminates bespoke N×M integrations. - Connectors are reusable and shareable across clients. - Open and vendor-neutral. - Cleaner separation between agent logic and integrations. ### Risks - Server access expands the attack surface; permissions matter. - Untrusted servers can attempt prompt injection or data exfiltration. - A young standard, still evolving. - Operational overhead of running and securing servers. ### Tools & technologies - MCP reference SDKs - Claude Desktop / Code - IDE MCP clients - Community MCP server registries ### Examples - An MCP server exposing a company knowledge base as readable resources to an agent. - A filesystem or database MCP server giving an agent scoped, permissioned access. - A shared connector for a SaaS API reused across several internal agents. ### FAQs **Who created MCP?** Anthropic introduced MCP as an open standard in late 2024, and it has since seen adoption across multiple vendors and tools. **Is MCP the same as function calling?** No. Function calling lets a model invoke tools; MCP standardizes how those tools, data and prompts are exposed and discovered across applications. **Why does MCP matter for the enterprise?** Integration is usually the bottleneck for agents. MCP turns one-off connectors into a reusable, governable ecosystem. **What are the security considerations?** Each server is an access point. Apply least-privilege permissions, vet servers, and treat their outputs as untrusted input subject to prompt-injection risk. ### References - Model Context Protocol — Official site & specification — https://modelcontextprotocol.io - Anthropic — Introducing the Model Context Protocol (2024) — https://www.anthropic.com/news/model-context-protocol Related: ai-agent, agentic-ai, enterprise-rag, harness-engineering --- ## What is a Multi-Agent Architecture? URL: https://santismm.com/en/knowledge/multi-agent-architecture | API: https://santismm.com/api/knowledge/multi-agent-architecture Category: architecture | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper **Definition.** A multi-agent architecture is a system design in which multiple specialized AI agents coordinate — through an orchestrator, a pipeline or peer interaction — to accomplish a task that is decomposed across them. A multi-agent architecture divides a task among several specialized agents that collaborate, delegate or compete to reach a goal, instead of relying on one general agent. Common shapes include an orchestrator that delegates to workers, pipelines where each agent owns a stage, and debate or critic patterns. It can improve modularity and reliability for complex tasks, but adds coordination overhead and should be adopted only when a single agent demonstrably falls short. ### Key takeaways - Several specialized agents beat one generalist for some complex tasks. - Common patterns: orchestrator-workers, pipelines, debate/critic. - Specialization improves modularity and focus per role. - Coordination, latency and cost overhead are the main costs. - Default to a single agent; go multi-agent only when measurement justifies it. ### Context As tasks grow, a single agent's context and reasoning get stretched. Splitting the work into focused roles — researcher, writer, reviewer; or planner and executors — can make each part more reliable and easier to evaluate. But multi-agent is not automatically better. Every added agent adds communication, failure modes and cost. The discipline is to decompose only where roles are genuinely separable and a single agent measurably underperforms. ### Architecture Orchestrator-workers: a lead agent plans and delegates subtasks to worker agents, then synthesizes results. Pipeline: agents are arranged in stages, each transforming the output of the previous. Peer patterns: agents debate, critique or vote to improve quality. Cross-cutting concerns — shared memory, message passing, error handling, budgets and observability — are where most multi-agent systems succeed or fail. Clear contracts between agents matter more than clever role names. ### Components - Orchestrator / lead agent - Worker / specialist agents - Shared memory & state - Message passing - Tools (often via MCP) - Guardrails & budgets - Observability ### Benefits - Modular, specialized roles that are easier to evaluate. - Parallelism for independent subtasks. - Separation of concerns across complex workflows. - Critic/debate patterns can raise output quality. ### Risks - Coordination overhead and added latency. - More failure modes and harder debugging. - Higher token cost from inter-agent communication. - Premature complexity when one agent would suffice. ### Tools & technologies - LangGraph - CrewAI - AutoGen - OpenAI Agents SDK - Model Context Protocol (MCP) ### Examples - An orchestrator delegating research, drafting and review to specialist agents. - A pipeline that extracts, transforms and validates data across stages. - A critic agent reviewing another agent's output before it is finalized. ### FAQs **Is multi-agent always better than a single agent?** No. It adds coordination, cost and failure modes. Prefer a single agent and adopt multi-agent only when a task is clearly separable and a single agent underperforms. **What is the orchestrator-workers pattern?** A lead agent plans a task, delegates subtasks to specialized worker agents, and synthesizes their results into a final answer. **How do multi-agent systems fail?** Through unclear contracts between agents, lost context, runaway loops, and compounding errors — which is why budgets and observability are essential. **How does MCP relate to multi-agent systems?** MCP standardizes how each agent connects to tools and data, making integrations reusable across the agents in the system. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - Yao et al. — ReAct (2022) — https://arxiv.org/abs/2210.03629 Related: agentic-ai, ai-agent, harness-engineering, model-context-protocol --- ## What is Prompt Engineering? URL: https://santismm.com/en/knowledge/prompt-engineering | API: https://santismm.com/api/knowledge/prompt-engineering Category: concept | Updated: 2026-06-21 | Version: 1.0 Evidence: benchmark | Confidence: high | Source: benchmark, paper, industry_observation **Definition.** Prompt engineering is the practice of designing and refining the instructions, context and examples given to a language model to reliably elicit a desired output. Prompt engineering is the practice of designing the inputs given to a language model so it produces the desired output reliably. A good prompt specifies the role, the task, the constraints, the output format and, when useful, examples. It is the most accessible lever for steering model behavior — and one layer of the broader harness around a model — but on its own it does not make a system reliable at scale. ### Key takeaways - A strong prompt states role, task, constraints, format and examples. - Examples (few-shot) usually beat instructions alone for structured tasks. - Chain-of-thought prompting improves multi-step reasoning. - Prompts should be tested and versioned, not hand-tuned by feel. - It is one layer of the harness, not a substitute for tools, memory and evaluation. ### Context Because models follow instructions in natural language, the way a task is phrased materially changes the result. Prompt engineering is the discipline of phrasing it well: being explicit about the goal, the audience, the constraints and the format you want back. It is the fastest, cheapest way to improve output quality, which is why it is where most teams start. But as systems grow into agents, prompting becomes one component among tools, memory, retrieval and evaluation — the full harness. ### Architecture Common techniques: zero-shot (instruction only), few-shot (instruction plus examples), chain-of-thought (ask for step-by-step reasoning), role and format specification, and decomposition (breaking a task into smaller prompts). Mature practice treats prompts as code: stored, versioned, tested against evals, and changed deliberately. Reusable prompt templates and structured output schemas reduce variance. ### Components - Role / persona - Task instruction - Constraints - Output format - Examples (few-shot) - Reasoning cues ### Benefits - Fastest, cheapest way to change model behavior. - No training or infrastructure required. - Works across models and tasks. - Easy to iterate and combine with other techniques. ### Risks - Fragile: small wording changes can shift behavior. - Prompt injection when prompts include untrusted input. - Hard to scale reliability by prompting alone. - Hidden coupling to a specific model's quirks. ### Tools & technologies - Prompt templates - Structured output / JSON schema - LangSmith / Langfuse (prompt testing) - Evaluation suites ### Examples - Adding a few worked examples to make a model output consistent JSON. - Asking for step-by-step reasoning to improve a math or logic answer. - Specifying a strict format so downstream code can parse the response. ### FAQs **Is prompt engineering still relevant as models improve?** Yes, but its role narrows. Better models need less coaxing, yet clear instructions, examples and format specs still measurably improve reliability — especially inside agents. **What is the difference from context engineering?** Prompt engineering focuses on the instruction. Context engineering is the broader task of deciding what information enters the model's limited context window at each step. **Does chain-of-thought always help?** It helps most on multi-step reasoning tasks, at the cost of more tokens. For simple lookups it adds latency without benefit. **How do you keep prompts reliable?** Treat them as code: version them, test them against evals, and change them deliberately rather than by trial and error. ### References - Wei et al. — Chain-of-Thought Prompting Elicits Reasoning in LLMs (2022) — https://arxiv.org/abs/2201.11903 - Anthropic — Prompt engineering overview — https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview Related: context-engineering, tool-use, agentic-ai, harness-engineering --- ## What is Prompt Injection? URL: https://santismm.com/en/knowledge/prompt-injection | API: https://santismm.com/api/knowledge/prompt-injection Category: governance | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper **Definition.** Prompt injection is a security attack where adversarial instructions embedded in untrusted input cause a language model to deviate from its intended behavior, bypass safeguards, or perform unintended actions. Prompt injection is an attack in which malicious instructions hidden in the input to a language model hijack its behavior — making it ignore its rules, leak data or misuse tools. It tops the OWASP Top 10 for LLM applications. The root cause is that models cannot reliably separate trusted instructions from untrusted content, so any text an agent reads — a web page, a document, a tool result — can carry an attack. ### Key takeaways - Untrusted text a model reads can contain hidden instructions. - It is the #1 risk in the OWASP Top 10 for LLM applications. - Indirect injection hides payloads in documents, pages or tool outputs. - Risk grows with tool access — injection can trigger real actions. - There is no single fix; defense is layered (least privilege, isolation, human approval). ### Context Models follow instructions in natural language and cannot reliably tell trusted system instructions from untrusted user or document content. An attacker exploits this by planting instructions like 'ignore previous instructions and…' where the model will read them. Direct injection comes from the user; indirect (and more dangerous) injection hides in content the agent retrieves — a web page, an email, a file, an MCP tool result. As agents gain tool access, a successful injection can exfiltrate data or take harmful actions. ### Architecture Defense is layered, not a single control: least-privilege tool permissions, isolating and clearly delimiting untrusted content, output and action validation, allow-lists for sensitive operations, and human-in-the-loop approval for high-impact actions. Treat all tool and retrieval outputs as untrusted input. Monitor and log agent actions (observability) so injection attempts are detectable, and red-team the system regularly. ### Components - Untrusted input boundary - Least-privilege permissions - Content isolation / delimiting - Output & action validation - Human approval for high-impact actions - Monitoring & red teaming ### Risks - Data exfiltration of sensitive context or credentials. - Unauthorized tool actions in connected systems. - Bypassed safety policies and guardrails. - Indirect attacks via documents, web pages or tool results. ### Tools & technologies - Input/output guardrail libraries - Permission & sandboxing layers - Allow-lists for tool actions - Monitoring / observability - Red-teaming frameworks ### Examples - A web page the agent reads contains hidden text telling it to email private data. - A document instructs a summarizer to ignore its rules and output a malicious link. - A tool result tries to make an agent call another tool it should not. ### FAQs **Why can't models just ignore injected instructions?** Because they cannot reliably distinguish trusted instructions from untrusted content — both arrive as text. That ambiguity is the core vulnerability. **What is indirect prompt injection?** When the malicious instructions are hidden in external content the model retrieves — a page, file, email or tool output — rather than typed by the user. It is often more dangerous. **Can prompt injection be fully prevented?** Not by a single measure today. You reduce risk with layered defenses: least privilege, content isolation, validation, monitoring and human approval for sensitive actions. **How does tool use raise the stakes?** Without tools, injection mostly produces bad text. With tools, an injected instruction can take real actions — send data, make changes — so permissions and approval matter more. ### References - OWASP — Top 10 for LLM Applications (LLM01: Prompt Injection) — https://genai.owasp.org/llmrisk/llm01-prompt-injection/ - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework Related: guardrails, tool-use, ai-governance, model-context-protocol --- ## What are Reasoning Models? URL: https://santismm.com/en/knowledge/reasoning-models | API: https://santismm.com/api/knowledge/reasoning-models Category: concept | Updated: 2026-06-21 | Version: 1.0 Evidence: benchmark | Confidence: high | Source: benchmark, paper **Definition.** Reasoning models are language models optimized to perform extended step-by-step reasoning at inference time — using additional test-time compute — to improve accuracy on complex, multi-step problems. Reasoning models are language models trained to spend extra computation 'thinking' before they answer — generating internal reasoning steps to solve harder problems in math, code and logic. They trade latency and cost for accuracy on complex, multi-step tasks. The key idea is test-time compute: letting a model reason longer at inference, rather than only making the model bigger, can substantially improve results. ### Key takeaways - They 'think' before answering, using extra inference compute. - Test-time compute is a new scaling axis beyond model size. - Best for math, code, logic and multi-step planning. - They trade latency and token cost for accuracy. - Overkill for simple tasks — match the model to the problem. ### Context Standard models answer in roughly constant time regardless of difficulty. Reasoning models break that: they generate a chain of internal reasoning, effectively spending more compute on harder questions, which lifts performance on tasks that need multi-step deduction. This introduced a second scaling axis. Beyond making models larger (train-time compute), you can let them reason longer at inference (test-time compute) — a major driver of recent progress on hard benchmarks. ### Architecture Reasoning models are typically trained to produce long internal reasoning before a final answer, often reinforced with reinforcement learning that rewards correct outcomes. At inference, more 'thinking' tokens generally mean better answers on hard problems. In agentic systems, reasoning models serve as strong planners and decision-makers, while cheaper, faster models can handle routine steps. Routing between them by task difficulty is a common cost-control pattern. ### Components - Extended reasoning (thinking tokens) - Test-time compute budget - RL-based training for reasoning - Final-answer extraction ### Benefits - Higher accuracy on complex, multi-step problems. - Strong at math, coding and planning. - Reasoning effort can be scaled per query. - Good planners at the core of capable agents. ### Risks - Higher latency and token cost. - Overkill — and wasteful — for simple tasks. - Longer reasoning is not always more correct. - Internal reasoning can be hard to audit or trust verbatim. ### Tools & technologies - Reasoning model tiers from major providers - Adjustable reasoning-effort settings - Model routing by task difficulty - Evaluation suites ### Examples - Solving a multi-step math or logic problem that trips up a standard model. - Planning a complex agent task before execution. - Routing only hard tickets to a reasoning model to control cost. ### FAQs **How are reasoning models different from standard LLMs?** They are trained and configured to reason at length before answering, spending more inference compute on hard problems instead of replying in near-constant time. **What is test-time compute?** Computation spent at inference (the model 'thinking' longer), as opposed to train-time compute spent making the model. It is a distinct way to improve results. **Should I always use a reasoning model?** No. They cost more and add latency. Use them for hard, multi-step problems and route simpler tasks to faster, cheaper models. **Do they eliminate hallucination?** No. Reasoning improves accuracy on many tasks but does not guarantee correctness; grounding, tools and evaluation remain necessary. ### References - Wei et al. — Chain-of-Thought Prompting Elicits Reasoning in LLMs (2022) — https://arxiv.org/abs/2201.11903 - DeepSeek-AI — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL (2025) — https://arxiv.org/abs/2501.12948 Related: foundation-models, agentic-ai, prompt-engineering, agentic-evaluation --- ## What is Tool Use (Function Calling)? URL: https://santismm.com/en/knowledge/tool-use | API: https://santismm.com/api/knowledge/tool-use Category: concept | Updated: 2026-06-24 | Version: 1.1 Evidence: production | Confidence: medium | Source: production_system, personal_experience, industry_observation, paper **Definition.** Tool use (function calling) is the capability that lets a language model call predefined external functions or APIs, with arguments it generates, and incorporate the results into its response or next step. Tool use, also called function calling, lets a language model invoke external functions, APIs or code to fetch information or take actions in the real world. The model decides which tool to call and with what arguments; the application runs the tool and returns the result, which the model uses to continue. Tool use is the bridge that turns a text generator into an agent that can actually do things. ### Key takeaways - Tool use connects a model to live data and real actions. - The model picks the tool and arguments; the app executes it. - Clear, well-described tools dramatically improve reliability. - It is the core mechanism behind agents and MCP. - Tool access expands capability and the security surface alike. ### Context On its own a model only produces text. Tool use breaks that boundary: given a set of declared tools, the model can choose to call one — for example a search, a database query or a payment API — and then reason over the result. How tools are described matters as much as the model. Tools written for a model to use — clear names, precise parameters, helpful descriptions and error messages — are a central concern of harness engineering. ### Architecture The loop: the application declares tools (name, description, parameter schema); the model emits a structured tool call; the application validates and executes it; the result returns to the model as an observation; the model continues or answers. MCP standardizes how tools are exposed and discovered across applications, so a tool written once can be reused by any compliant client. Guardrails and permissions wrap execution to keep it safe. ### Components - Tool declaration (schema) - Tool selection (model) - Argument generation - Execution layer - Result / observation - Guardrails & permissions ### Benefits - Grounds answers in live, real data. - Lets models take real actions, not just describe them. - Extends a model without retraining. - Composes into full agentic workflows. ### Risks - Prompt injection can trigger unintended tool calls. - Wrong arguments or tool misuse cause real-world errors. - Over-broad tool access widens the attack surface. - Latency and cost grow with each tool round-trip. ### Tools & technologies - Function calling APIs - Model Context Protocol (MCP) - LangGraph / Agents SDKs - Schema validation (e.g. JSON Schema, Zod) ### Examples - A model calling a weather API to answer a forecast question. - An agent querying a database to look up an order before acting. - A coding agent invoking a test runner and reading the results. - In practice: an autonomous OpenClaw agent issued 800 tool calls across 10 distinct tools over 57 days — exec (481), web_search (190), write (48), web_fetch (22), read (21) and others — averaging 5.97 tool calls per session. A single-operator local-first deployment, measured from the agent's own traces. ### FAQs **Is tool use the same as MCP?** No. Tool use is the model capability to call functions. MCP is a standard for how those tools and data are exposed and discovered across applications. **How do you make tool use reliable?** Write tools for the model: clear names, precise parameter schemas, useful descriptions and informative errors. Validate arguments and constrain permissions. **What are the security risks?** Tools are real access. Prompt injection can attempt to trigger harmful calls, so apply least-privilege permissions, validate inputs, and treat tool outputs as untrusted. **Does tool use make a model an agent?** It is the key enabler. An agent combines tool use with a control loop, memory and a goal so it can act over multiple steps. ### References - Schick et al. — Toolformer: Language Models Can Teach Themselves to Use Tools (2023) — https://arxiv.org/abs/2302.04761 - Anthropic — Tool use (function calling) — https://docs.anthropic.com/en/docs/build-with-claude/tool-use Related: ai-agent, model-context-protocol, agentic-ai, harness-engineering ================================================================= # ENTERPRISE AI PATTERNS LIBRARY ================================================================= ## Context Compression URL: https://santismm.com/en/patterns/context-compression | API: https://santismm.com/api/patterns/context-compression Category: cost | Updated: 2026-06-24 | Version: 1.1 Evidence: production | Confidence: low | Source: production_system, personal_experience, industry_observation Context compression reduces the tokens fed to a model on each call while preserving the information it actually needs to act. Use it on long-running agents and long conversations to cut cost and latency and to stay inside the context window. The three levers are summarizing history, pruning irrelevant context, and compressing prompts. The central risk is lossy: dropping the one detail that mattered. Measure information retained, not just tokens saved. ### Problem Long-running agents and multi-turn conversations accumulate context: every tool result, prior message, and retrieved document is replayed on the next call. Token count grows roughly linearly with the interaction, so per-call cost and latency climb, and eventually the window overflows and the oldest (sometimes most important) content is silently truncated. Naive fixes — bigger windows, more aggressive truncation — either raise cost or destroy the information the model needs to stay coherent. ### When to use it Applies when context grows unbounded relative to what any single step needs: conversational assistants with long histories, autonomous agents looping over many tool calls, RAG pipelines that over-retrieve, and batch jobs where prompt size dominates cost. It fits when much of the accumulated context is redundant or stale, when you control prompt assembly, and when you can tolerate some reconstruction error. It is a poor fit when every token is load-bearing (legal, audit, exact-recall tasks) or when interactions are short enough that the window is never pressured. ### Solution Treat the live context as a budget you actively manage rather than an append-only log. Three complementary levers exist. Summarization replaces a span of history with a shorter synopsis — typically a rolling summary of older turns, refreshed periodically, while recent turns stay verbatim. Pruning removes context that is irrelevant to the current step: deduplicate, drop stale tool output, and select only the retrieved chunks that score above a relevance threshold. Prompt compression (for example LLMLingua) uses a smaller model to delete or rephrase low-information tokens before sending the prompt, trading a small accuracy cost for large token reductions. Compose these into a pipeline with explicit boundaries: keep a verbatim recent window, a rolling summary of older history, and a retrieval slot filled on demand. Protect a 'pinned' region for facts that must never be compressed — identifiers, constraints, the current goal. Crucially, instrument the result: run an evaluation set comparing answers with and without compression so you can see when quality degrades, and tune the aggressiveness per workload rather than globally. Compression is a quality-versus-cost dial, not a free win. ### Components - Rolling summarizer - Relevance pruner - Prompt compressor - Pinned region - Context budget controller - Retention evaluator ### Benefits - Sending fewer tokens directly reduces input cost on every call, which compounds across long agent loops and high-volume traffic. - Smaller prompts mean less to encode and shorter time-to-first-token, improving responsiveness in interactive and agentic flows. - Bounding live context lets long conversations and many-step agents continue without overflowing the window or silently truncating. - Removing redundant and stale context can improve quality by reducing distraction, helping the model attend to what currently matters. ### Risks - Summaries and pruning can discard the single detail that later turns out to be decisive, producing confidently wrong answers. - Rolling summaries summarize prior summaries; small omissions compound over many cycles until the thread quietly drifts. - Running a summarizer or compressor adds its own latency, cost, and failure surface, which can offset savings on short interactions. - Aggressive eviction can silently remove constraints or instructions the model still depends on, with no obvious error signal. ### When not to use it - When every token is load-bearing — legal, audit, compliance, or precise data extraction — lossy compression is unacceptable. - If conversations rarely pressure the window, compression overhead costs more than it saves and adds needless complexity. - Without a retention evaluation harness, deploy nothing: you cannot tell whether compression is silently degrading answers. ### Technologies - Summarization - Context pruning - RAG - Prompt compression (LLMLingua) ### Examples - An agent iterating over a large codebase keeps a verbatim recent window plus a rolling summary of earlier steps, pinning the task spec and file paths so it does not lose the goal. - A multi-session support bot summarizes prior turns into a compact case summary, pruning resolved sub-issues while pinning the customer's account constraints. - A retrieval pipeline that fetches many chunks applies relevance pruning and prompt compression to send only high-signal passages, cutting tokens without losing the answer. ### Production evidence - Context: Single-operator, local-first OpenClaw deployment observed over 57 days (161 sessions / 2,776 turns), aggregated from the agent's own trajectory traces. - Scenario: Long autonomous transcripts are compacted preemptively and tool results truncated to stay within the prompt budget. - Technology: Preemptive compaction with a safety margin, tool-result truncation, a context-pruning hook, and a midturn overflow precheck. - Load: 2,810 context-compiled events across 2,776 turns over 57 days. - Results: Compaction ran inline throughout the window, keeping multi-step autonomous turns within budget (p95 87.6s per turn) without context-overflow failures surfacing. Single-operator local-first deployment. ### KPIs - Tokens per call (input): The primary cost driver. Track the distribution before and after compression; a healthy result is a clear reduction with no rise in downstream errors. - Information retention / task quality: Compare answers with and without compression on an eval set. Good looks like quality holding steady within your tolerance as tokens drop. - End-to-end latency: Net of compression overhead. Good is lower total latency; watch that summarizer or compressor calls do not erase the savings. - Context-overflow / truncation rate: How often interactions hit the window limit. Good is driving this toward zero without resorting to dropping pinned content. ### Observed failure modes - A summary omits a constraint mentioned early; many turns later the agent violates it because that fact is simply gone from context. - Repeated re-summarization amplifies paraphrase errors and omissions until the running summary no longer reflects what actually happened. - A misconfigured budget compresses identifiers or instructions that were meant to be protected, breaking correctness silently. - An aggressive relevance threshold filters out context that mattered for an edge case, so quality looks fine in tests but fails in the field. ### Lessons learned - Token reduction is trivial to maximize and meaningless alone; the real metric is whether the model still answers correctly. - Explicitly protect identifiers, constraints, and the current goal so no compression stage can evict them. - Compress old history, not the active context; the most recent exchanges carry the most decision-relevant signal. - Tune aggressiveness per workload against an eval set; what is safe for chit-chat is reckless for an audit task. ### FAQs **How is this different from long-term memory?** Long-term memory persists facts outside the prompt and retrieves them on demand; context compression shrinks the live context sent on each call. They are complementary: memory decides what to bring back, compression decides how compactly it sits in the window. **Summarize, prune, or compress — which should I use?** Prune first (free, lossless when removing true redundancy), summarize older history when it grows unbounded, and add prompt compression only when you still need more headroom and can validate the quality cost. Most systems combine all three. **How do I know compression is hurting quality?** Run an evaluation set with compression on and off and compare task outcomes, not just token counts. Watch for confidently wrong answers and dropped constraints — those are the signature of lossy compression that has gone too far. ### References - Jiang et al. — LLMLingua (2023) — https://arxiv.org/abs/2310.05736 - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents Related: long-term-memory, semantic-caching --- ## Evaluator-Optimizer URL: https://santismm.com/en/patterns/evaluator-optimizer | API: https://santismm.com/api/patterns/evaluator-optimizer Category: reliability | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper One LLM generates a response while a second LLM evaluates it against criteria and returns feedback; the generator revises and the loop repeats until the evaluation passes. It raises quality on tasks with clear evaluation criteria, at the cost of extra calls. ### Problem A single-pass output may miss requirements, and there is no built-in mechanism to check and improve it before it is used. ### When to use it Use evaluator-optimizer when you can articulate clear evaluation criteria and iterative refinement measurably improves the result — for example translation quality, code that must pass tests, or writing against a rubric. ### Solution A generator produces a candidate; an evaluator (a separate LLM call or a deterministic check) scores it against explicit criteria and returns actionable feedback. The generator revises, and the cycle repeats until criteria are met or a budget is reached. Separating generation from evaluation mirrors how a human writer benefits from an editor: the critic catches issues the author misses, and explicit criteria keep the loop converging. ### Components - Generator - Evaluator (LLM judge or rule check) - Explicit criteria - Revision loop - Stop condition / budget ### Benefits - Higher quality on tasks with clear criteria. - Catches errors a single pass would ship. - Feedback is explicit and actionable. ### Risks - Extra calls add latency and cost. - A weak evaluator gives misleading feedback. - Loops can fail to converge without a budget. ### When not to use it - When criteria cannot be clearly defined. - When a single pass is already good enough. - When latency or cost budgets are very tight. ### Technologies - LangGraph - LLM-as-judge - OpenAI Agents SDK - Evaluation suites ### Examples - Generating code, running tests, and revising until they pass. - Drafting a translation and refining it against the source. - Writing to a rubric with a critic enforcing each criterion. ### KPIs - Acceptance rate: Share of candidate outputs the evaluator accepts on first pass — too high means the bar is too low, too low means the generator or rubric is off. - Iterations to accept: Average evaluate→revise loops before acceptance; rising counts flag a weak generator or vague criteria. - Cost & latency per accepted output: Total tokens and wall-clock across all loop iterations, not just the final call — the loop multiplies both. - Eval–human agreement: How often the evaluator's verdict matches a human reviewer on a sampled set; the loop is only as good as the evaluator. ### Observed failure modes - Reward hacking: the generator learns to satisfy the evaluator's wording rather than the real goal. - Weak or miscalibrated evaluator: it accepts bad outputs or rejects good ones, so the loop adds cost without quality. - Infinite or oscillating loops when no candidate ever clears the bar — without an iteration cap the cost is unbounded. - Criteria drift: vague or shifting rubrics make acceptance non-deterministic and hard to audit. ### Lessons learned - Cap iterations and define a fallback (return best-so-far, or escalate) so the loop always terminates. - Make acceptance criteria explicit and stable; an evaluator is only as good as its rubric. - Validate the evaluator against human judgement before trusting it as a gate. - Use the loop only where quality justifies the multiplied cost — not for cheap, low-stakes outputs. ### FAQs **How is this different from reflection?** Reflection has the same model self-critique. Evaluator-optimizer separates the roles: a distinct evaluator judges the generator, which often gives sharper, less biased feedback. **Can the evaluator be deterministic?** Yes. For code, a test runner is an ideal evaluator; for structured output, a schema check works. Use a model judge for nuanced criteria. **How many iterations?** Set a budget (e.g. 2–3) and stop when criteria pass. Unbounded loops waste cost and may not converge. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents Related: reflection, prompt-chaining, orchestrator-workers --- ## Goal Decomposition URL: https://santismm.com/en/patterns/goal-decomposition | API: https://santismm.com/api/patterns/goal-decomposition Category: orchestration | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper Goal decomposition has an agent break a high-level goal into an ordered set of smaller, tractable sub-tasks — a plan — before acting, then execute and monitor that plan, re-planning when steps fail. The explicit plan becomes an inspectable artifact you can review, gate, and debug. Use it when a goal needs several dependent steps and reactive, step-at-a-time agents drift or stall; skip it for simple, single-shot tasks. ### Problem A single LLM call handed a broad, multi-step goal tends to improvise. Reactive agents that choose one action at a time can lose the thread on long horizons: they repeat work, skip prerequisites, or chase a dead end without realizing the overall objective is now unreachable. Because no plan exists as an artifact, you cannot review intended steps before they run, cannot tell whether a failure came from a bad strategy or a bad execution, and cannot easily resume after an interruption. The agent's reasoning is implicit, transient, and hard to audit. ### When to use it This pattern fits goals that decompose into multiple interdependent steps with a meaningful ordering — research-then-synthesize, migrate-then-verify, gather-then-reconcile-then-report. It assumes the model can produce a reasonable plan from the goal and available tools, and that steps are observable enough to detect failure. It is most valuable where steps are costly, side-effecting, or hard to undo, so reviewing the plan before execution pays off. It is a poor fit when the next action is obvious from the current state, or when the environment changes so fast that any upfront plan is stale before the second step. ### Solution Split the agent into a planning phase and an execution phase. The planner reads the goal, the available tools, and the current state, and emits an explicit, ordered plan: a list (or graph) of sub-tasks with their dependencies and expected outputs. Treating the plan as a first-class artifact is the core idea — it can be logged, shown to a human for approval, scored against policy, and diffed across runs. Encode dependencies explicitly so independent sub-tasks can run in parallel and dependent ones wait for their inputs, rather than forcing a brittle linear sequence the model invented. An executor then runs the plan step by step, feeding each step's result forward and checking it against the step's expected output. When a step fails, returns something unusable, or invalidates a downstream assumption, hand control back to the planner to re-plan from the current state instead of blindly continuing — this closed loop is what separates robust decomposition from one-shot planning. Keep plans as shallow as the goal allows: prefer a few well-chosen steps over a deep tree, gate re-planning with a budget so the agent cannot loop forever, and let trivial goals bypass planning entirely. ### Components - Planner that emits an ordered, dependency-aware plan - Plan representation (task list or task graph) as an inspectable artifact - Executor that runs steps and forwards results - Per-step verification against expected outputs - Re-planning trigger and loop with a step/iteration budget - Optional human approval gate before execution ### Benefits - Long-horizon goals stay coherent because intended steps are decided up front, not improvised one at a time. - The explicit plan is inspectable: it can be reviewed, approved, audited, and diffed before any side effect runs. - Failures are easier to localize — a bad plan is distinguishable from a bad step execution. - Independent sub-tasks expose parallelism and let work resume from the last completed step after an interruption. ### Risks - A flawed initial decomposition propagates: every downstream step inherits a wrong assumption or missing prerequisite. - Over-planning adds latency and cost on simple goals that a reactive agent would finish in one step. - Plans go stale in fast-changing environments, so executing a step still based on an outdated world state. - Unbounded re-planning loops where the agent repeatedly rewrites the plan without making real progress. ### When not to use it - The next action is obvious from the current state and a single reactive step solves the goal. - The environment changes faster than a plan stays valid, making any upfront sequence stale. - Steps are cheap, reversible, and independent, so the overhead of planning outweighs its benefit. ### Technologies - Planner/executor frameworks - LangGraph - ReAct / Plan-and-Solve - Task graphs ### Examples - A research assistant plans gather-sources, extract-claims, cross-check, then synthesize, running the source gathering in parallel before the dependent synthesis step. - A code-migration agent plans inventory-usages, transform-files, run-tests, then re-plans the transform step when the test step surfaces a missed edge case. - A data-reconciliation agent decomposes a 'close the books' goal into pull-ledgers, normalize, match-entries, and flag-exceptions, with matching gated behind successful normalization. ### KPIs - Goal completion rate: Share of goals fully achieved end-to-end; good looks like decomposition beating a reactive baseline on the same multi-step tasks. - Re-plan frequency: How often a run triggers re-planning; a healthy band means the loop catches real failures without thrashing on every step. - Steps per goal vs. minimum: Plan length relative to a sensible minimum; watch for over-planning that inflates steps on simple goals. - Plan-approval pass rate: Fraction of plans accepted by reviewers or policy checks before execution; low rates signal systematically weak decomposition. ### Observed failure modes - Bad decomposition propagates: a wrong early assumption corrupts every dependent step downstream. - Re-planning loop: the agent rewrites the plan repeatedly without converging or making progress. - Stale plan execution: a step runs against a world state that changed since the plan was made. - Over-decomposition: a trivial goal is split into needless steps, adding latency, cost, and failure surface. ### Lessons learned - Make the plan a real artifact — log it, show it, diff it — so failures are debuggable rather than mysterious. - Always close the loop: detect step failure and re-plan from current state instead of continuing blindly. - Bound both plan depth and re-planning with explicit budgets to prevent shallow goals from spiraling. - Let trivial goals skip the planner; reserve decomposition for genuinely multi-step, dependent work. ### FAQs **How is this different from a reactive ReAct-style agent?** A reactive agent decides one action at a time from the current state, with no plan as an artifact. Goal decomposition commits to an ordered plan up front, making intended steps inspectable and dependency ordering explicit. In practice the two are often combined: plan first, then execute reactively within each step and re-plan when a step fails. **What happens when a step fails mid-plan?** Hand control back to the planner to re-plan from the current state rather than continuing blindly. The closed re-planning loop is what makes decomposition robust. Bound it with a budget so a persistently failing step cannot trigger endless rewrites without progress. **When does planning hurt more than it helps?** On simple, single-step goals where the next action is obvious, or in environments that change faster than a plan stays valid. There, upfront planning adds latency and a stale-plan risk. Detect trivial goals and let them bypass the planner, reserving decomposition for genuinely multi-step, dependent work. ### References - Yao et al. — ReAct (2022) — https://arxiv.org/abs/2210.03629 - Wang et al. — Plan-and-Solve Prompting (2023) — https://arxiv.org/abs/2305.04091 Related: supervisor-agent, orchestrator-workers, task-prioritization --- ## Human Approval Gate URL: https://santismm.com/en/patterns/human-approval-gate | API: https://santismm.com/api/patterns/human-approval-gate Category: safety | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper **Definition.** A human approval gate is a control checkpoint inserted before a high-impact automated action, where a person reviews and approves, edits or rejects the proposed action before it executes. A human approval gate pauses an automated workflow at a defined checkpoint so a person can review, edit or reject a proposed action before it executes — especially for high-impact, irreversible or regulated operations. It is the operational form of human-in-the-loop oversight. ### Problem Letting an AI system execute high-impact actions autonomously risks costly, irreversible or non-compliant mistakes with no chance for human judgment. ### When to use it Use an approval gate for actions whose cost of error outweighs the latency of review: payments, deletions, external communications, production changes, or anything regulated. ### Solution Insert a checkpoint before the sensitive action: the system prepares the proposed action with enough context, then suspends and routes it to a human who approves, edits or rejects. On approval it proceeds; on timeout it falls back safely. Every decision is logged for audit. Gate only the high-impact steps, not everything — over-gating destroys the value of automation and causes approval fatigue. Choose checkpoints by risk. ### Components - Risk-based checkpoint - Proposed-action preview - Approve / edit / reject - Timeout & safe fallback - Audit log ### Benefits - Prevents costly or irreversible mistakes. - Keeps accountability with a human. - Satisfies compliance and oversight requirements. - Builds trust, enabling gradual autonomy. ### Risks - Adds latency and limits throughput. - Rubber-stamping if reviewers lack context or time. - Approval fatigue from too many gates. - Bottlenecks if reviewers are unavailable. ### When not to use it - For low-impact, easily reversible actions. - When throughput must be high and risk is low. - When a deterministic guardrail can safely auto-approve. ### Technologies - LangGraph (interrupts) - Workflow / approval systems - Audit logging ### Examples - An agent drafting a refund a human approves before it is issued. - A production change that pauses for sign-off before deploying. - An outbound email queued for review before sending. ### Production evidence - Context: Enterprise workflows where an agent can trigger irreversible or regulated actions — refunds, account changes, outbound communications, production deployments. - Scenario: The agent prepares the action with full context and pauses; a reviewer approves, edits or rejects it; on approval it executes, on timeout it falls back safely. Every decision is logged. - Technology: A workflow engine with interrupts (e.g. LangGraph), an approval queue/UI, and an audit log. - Load: Only a minority of high-impact steps are gated; the bulk of low-impact steps run automatically, so reviewer volume stays bounded. - Results: Observed pattern: irreversible errors are caught before execution and accountability stays with a human, at the cost of added latency on gated steps. Gate by risk and measure approval latency and rubber-stamp rate on your own workflow — these are reference observations, not guaranteed numbers. ### KPIs - Approval latency: Time an action waits at the gate; the core cost of the pattern and the first thing to watch for bottlenecks. - Rejection / edit rate: Share of proposals a human rejects or edits — near-zero often means rubber-stamping, very high means the agent isn't trusted yet. - Throughput vs. gated steps: Tasks completed per hour against how many steps are gated; over-gating collapses throughput. - Timeout / fallback rate: How often actions hit the timeout and take the safe fallback; a rising rate signals reviewer overload. ### Observed failure modes - Rubber-stamping: reviewers approve without real scrutiny when they lack context or time, defeating the gate. - Approval fatigue and bottlenecks from over-gating low-impact steps. - Silent auto-execution on timeout when no safe fallback is defined. - Insufficient context in the proposal, so the human can't make an informed decision. ### Lessons learned - Gate by risk, not by default — automate low-impact steps and reserve gates for irreversible or regulated actions. - Give reviewers enough context and a clear approve/edit/reject choice to prevent rubber-stamping. - Always define a safe fallback on timeout; never silently execute a gated action. - Log every decision for audit — the gate is also your compliance evidence. ### FAQs **How is this different from human-in-the-loop?** It is the concrete implementation of the human-in-the-loop principle: a specific approval checkpoint in a workflow before a sensitive action. **Won't approvals slow everything down?** Only if you over-gate. Apply gates by risk — automate low-impact steps and reserve approval for high-impact, irreversible or regulated actions. **What happens on timeout?** Define a safe fallback: hold the action, escalate, or cancel. Never silently auto-execute a gated action just because no one responded. ### References - European Union — AI Act, Article 14 (Human oversight) — https://artificialintelligenceact.eu/article/14/ - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework Related: reflection, evaluator-optimizer, prompt-chaining --- ## Human Escalation URL: https://santismm.com/en/patterns/human-escalation | API: https://santismm.com/api/patterns/human-escalation Category: safety | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper Hand the whole task to a human when the agent detects it is out of its depth — low confidence, repeated failure, ambiguity, or sensitive situations — and pass full context so the person can take over without re-investigating. Unlike an approval gate, which pauses one action for sign-off, escalation transfers ownership so the agent stops driving. The hard part is calibrating triggers to avoid both over- and under-escalation. ### Problem An autonomous agent will inevitably encounter cases it cannot handle well: inputs outside its training distribution, requests it keeps failing to satisfy, genuinely ambiguous goals, or emotionally and legally sensitive moments. If it presses on anyway, it produces confidently wrong answers, loops, or harmful actions — and the user discovers the failure too late. Yet routing everything to humans defeats the point of automation and overwhelms staff. The system needs a disciplined way to recognize the edge of its competence and transfer the task before damage is done. ### When to use it Use this pattern wherever an agent acts with meaningful autonomy and the cost of a wrong outcome exceeds the cost of a human glancing at it: customer support, claims and case handling, financial or medical triage, content moderation, and operational copilots. It assumes a human queue or on-call function exists to receive escalations and that the agent can observe signals about its own performance. It is most valuable when failures are silent — when a confidently wrong answer is worse than no answer — and when a subset of cases is known to be hard, rare, or regulated. ### Solution Define explicit escalation triggers and wire them into the agent's main loop as first-class exit conditions, not afterthoughts. Common triggers are confidence below a threshold (from model scores, self-critique, or a verifier), loop or repeated-failure detection (the agent retries the same step without progress), structural ambiguity (multiple valid interpretations of the goal), and sensitivity signals (negative sentiment, safety keywords, high-value accounts, or regulated topics). Each trigger should map to a routing decision: which human or team, with what priority. Treat thresholds as tunable parameters owned by the team, reviewed against real outcomes, because they encode the trade-off between automation rate and error rate. When a trigger fires, the agent must perform a clean handoff: stop acting, package the full context — original request, what it attempted, intermediate results, its current best guess, and why it escalated — and route it to the right queue via a ticket or live handoff. The receiving human should be able to take over without re-investigating from scratch; context quality is what makes escalation feel like help rather than a dropped ball. Always provide a graceful fallback message to the end user ("I'm bringing in a specialist") so the experience degrades smoothly. Finally, log every escalation with its trigger and resolution so appropriateness can be measured and triggers retuned. ### Components - Trigger evaluator - Context packager - Router - Handoff channel - Fallback responder - Audit log ### Benefits - Hard cases reach a human before the agent produces a confidently wrong outcome, capping the blast radius of mistakes. - Only genuinely difficult cases are handed off, so routine volume stays automated and staff focus on what needs judgment. - A clean handoff with context means users are helped rather than bounced, and humans resume without starting over. - Logged triggers and resolutions provide the evidence trail regulators and risk owners expect for meaningful human oversight. ### Risks - Thresholds set too conservatively push easy cases to humans, erasing automation gains and burying staff in noise. - Thresholds set too loosely let the agent power through cases it should have handed off, causing silent bad outcomes. - If the payload is thin, the human re-investigates from scratch and escalation feels like a dropped task, not assistance. - Model self-confidence often does not track real accuracy, so naive score thresholds escalate the wrong cases in both directions. ### When not to use it - If there is no staffed queue or on-call function to take over, escalation has nowhere to go; invest in a safe-stop or recovery path instead. - When you only need approval for one specific high-impact step while the agent keeps the task, use a human-approval gate, not full ownership transfer. - For cheap, easily reversible tasks where a wrong answer costs nothing, the overhead and latency of escalation outweigh the benefit. ### Technologies - Confidence scoring - Routing - Ticketing / handoff systems - Audit logging ### Examples - A support agent resolves routine questions but escalates to a human queue on detected frustration, repeated unhelpful answers, or account-sensitive requests, passing the full conversation. - An insurance agent auto-processes clear claims and escalates ambiguous, high-value, or fraud-flagged ones to an adjuster with its findings and the reason attached. - An autonomous coding agent that fails the same test repeatedly stops, summarizes what it tried and where it is blocked, and hands the task to an engineer instead of churning. ### KPIs - Escalation rate: Share of tasks handed to humans. Watch the trend and the distribution, not a target number — a sudden spike or drop signals a miscalibrated trigger or a shift in input mix. - Escalation appropriateness: Of escalated cases, how many genuinely needed a human (true positives) versus could have been handled. Sampled human review of escalations is the most reliable read. - Missed-escalation rate: Of automated resolutions, how many later turned out to be wrong and should have been escalated. The hardest and most important signal; mine complaints, reopens, and audits to find them. - Handoff context sufficiency: How often the receiving human can take over without re-contacting the user or re-investigating. Track via agent feedback on whether the package was complete. ### Observed failure modes - Triggers tuned once and never revisited fall out of step as inputs and models change, silently shifting the automation/error balance. - Cases route into a queue that no one owns or that is overwhelmed, so escalated users wait indefinitely — worse than a wrong answer. - An agent optimized to avoid escalation learns to express false confidence, suppressing the very signal the pattern depends on. - Handoff strips formatting, intermediate reasoning, or attachments, forcing the human to rebuild the situation and erasing the speed benefit. ### Lessons learned - Validate that your confidence signal correlates with actual accuracy before thresholding on it; pair model scores with a verifier or self-critique. - The difference between a good and bad escalation is almost entirely the handoff payload; invest there before tuning thresholds. - Treat thresholds as living parameters reviewed against sampled escalations and missed escalations, owned by the team, not frozen at launch. - Even a perfect trigger fails sometimes; a graceful holding message and an owned queue prevent failures from becoming abandonments. ### FAQs **How is this different from a human-approval gate?** An approval gate pauses one specific high-impact action and asks a human to sign off, then the agent continues. Escalation transfers ownership of the whole task — the agent stops driving because it shouldn't proceed at all. Use a gate for 'should I do this one thing?' and escalation for 'I'm out of my depth, please take over.' **What's the right escalation rate?** There is no universal number; it depends on task difficulty mix and the cost of errors. Optimize for appropriateness, not a target rate: escalate cases that genuinely need a human and minimize both unnecessary handoffs and missed escalations. Review the rate as a signal of miscalibration, not as a goal in itself. **Can I just escalate whenever model confidence is low?** It's a useful trigger but rarely sufficient alone, because model self-confidence often does not track real accuracy. Combine it with loop detection, ambiguity checks, and sensitivity signals, and validate that your confidence measure actually correlates with correct outcomes before trusting a threshold. ### References - EU AI Act — Article 14 (Human oversight) — https://artificialintelligenceact.eu/article/14/ - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework Related: human-approval-gate, recovery-strategy, routing --- ## Long-Term Memory URL: https://santismm.com/en/patterns/long-term-memory | API: https://santismm.com/api/patterns/long-term-memory Category: retrieval | Updated: 2026-06-24 | Version: 1.1 Evidence: production | Confidence: low | Source: production_system, personal_experience, industry_observation Give an agent persistent memory across sessions so it remembers facts, user preferences, and prior outcomes beyond a single context window. A write path decides what to store, summarizes it, and deduplicates it; a read path retrieves only the relevant memories into context when needed. Unlike semantic caching, which caches whole answers to skip recomputation, long-term memory stores durable facts and state and recomposes them into fresh reasoning each time. ### Problem The context window is finite and resets between sessions. An agent that only sees the current conversation forgets a user's stated preferences, decisions made last week, and the outcome of prior tasks. Stuffing all history into every prompt is impossible past a certain scale and degrades reasoning as the window fills with low-value tokens. Teams need a way to persist the small set of facts that matter and surface them precisely when they are relevant. ### When to use it Use this when an agent serves the same users or works on the same long-running tasks repeatedly: assistants that learn preferences, support agents that track a customer's history, coding agents that remember project conventions, or multi-step workflows spanning days. It assumes you can store data outside the model (a vector store, database, or memory framework) and that you control both when memories are written and how they are retrieved into the prompt. ### Solution Separate the write path from the read path. On the write path, after a turn or task completes, an extraction step decides what is worth remembering: stable facts, preferences, commitments, and outcomes — not transient chatter. Candidate memories are summarized into compact, self-contained statements, checked against existing memories to deduplicate and to detect contradictions, then written to a store with metadata: a memory type, a timestamp, a source, and the user or scope it belongs to. Writing less but writing well is the goal; noisy memories poison later retrieval. On the read path, before the agent reasons, you retrieve candidate memories relevant to the current task — typically by semantic similarity plus filters on scope and recency — rank them, and inject only the top few into context. Treat retrieval as a precision problem: a handful of correct memories beats a large, loosely related set. Distinguish memory types so retrieval can be targeted: episodic (what happened), semantic (durable facts and preferences), and procedural (how to do a recurring task). Periodically consolidate and expire memories so the store stays small, current, and free of contradictions. ### Components - Memory extractor (write path) - Deduplication and contradiction check - Memory store - Retriever (read path) - Context assembler - Consolidation and expiry job ### Benefits - The agent recalls preferences, decisions, and outcomes from prior sessions, so users do not have to repeat context and the agent behaves consistently over time. - Retrieving a few relevant memories keeps the window focused on high-value tokens instead of dumping full history, which preserves reasoning quality and reduces cost. - As stable facts and preferences accumulate, the agent tailors responses more accurately with each interaction without retraining the model. - Because memories live in an external store with metadata, you can inspect, correct, export, and delete what the agent knows — important for trust and compliance. ### Risks - Without consolidation and expiry, the store accumulates outdated facts and conflicting statements, and the agent confidently acts on the wrong one. - Persisting user data raises retention, consent, and access-control obligations; memories can leak sensitive information across sessions or users if scope is not enforced. - Low precision injects irrelevant or wrong memories that mislead reasoning; low recall silently drops the memory that mattered, making failures hard to diagnose. - Over-eager writing inflates the store, slows retrieval, raises storage and embedding costs, and dilutes the signal that good retrieval depends on. ### When not to use it - If sessions are independent and nothing needs to carry over, persistent memory adds complexity, cost, and privacy surface for no benefit. - When the goal is to reuse a previous answer for a repeated query, semantic caching is the right tool; long-term memory is for remembering facts and state, not caching outputs. - Where regulation or policy forbids retaining user data, do not persist memories; rely on in-session context or explicit, scoped storage the user controls. ### Technologies - Vector store - Memory frameworks (Mem0 / LangMem) - RAG - Summarization ### Examples - Across sessions it remembers tone, formats, recurring contacts, and standing instructions, retrieving the few that apply to the current request instead of re-asking. - On each contact it retrieves the customer's prior issues, entitlements, and resolutions scoped to that account, so it continues rather than restarts the conversation. - It stores procedural memories — build commands, naming rules, review preferences — and recalls them when working in the same repository over many sessions. ### Production evidence - Context: Single-operator, local-first OpenClaw deployment observed over 57 days (161 sessions / 2,776 turns), aggregated from the agent's own trajectory traces. - Scenario: The agent persists workspace memory files and per-session traces for cross-turn and cross-session continuity, with a semantic-recall plugin available on demand. - Technology: Workspace memory files (MEMORY.md, IDENTITY.md, SOUL.md, USER.md, HEARTBEAT.md), persistent session ids and lifecycle events, and an active-memory plugin (memory_search/get/recall). - Load: 134 persisted session files across the 57-day window; semantic recall invoked once. - Results: Continuity held across 134 persisted sessions over 57 days through structural workspace memory; explicit semantic recall was rarely needed (one call) in this autonomous workload. Single-operator local-first deployment. ### KPIs - Retrieval precision of injected memories: Of the memories placed in context, the share that were actually relevant. This is the metric that most directly governs answer quality; good looks like the injected set being almost entirely on-topic, with irrelevant memories rare. - Retrieval recall on memory-dependent tasks: On tasks that require a known stored fact, how often that fact is actually retrieved. Good looks like the right memory surfacing reliably; persistent misses point to extraction or indexing gaps. - Memory store size and growth rate: Total memories and how fast they accumulate per active user. Good looks like growth tracking genuinely new durable facts, not unbounded climb — a runaway curve signals over-eager writing. - Staleness and contradiction rate: Share of retrieved memories that are outdated or conflict with a newer truth. Good looks like a low and stable rate, evidence that consolidation and expiry are keeping pace with change. ### Observed failure modes - Writing everything turns the store into noise; retrieval then surfaces low-value or wrong memories. Fix by raising the bar for what gets written and reviewing extraction quality. - An old fact is retrieved and acted on after the truth changed, with no signal that it is outdated. Mitigate with timestamps, recency-weighted ranking, and explicit supersession on write. - A memory from one user, tenant, or project is retrieved into another's context because scope filters were missing or wrong — a privacy and correctness failure at once. - To compensate for poor ranking, teams inject many memories, refilling the window with marginal tokens and degrading the very reasoning memory was meant to support. ### Lessons learned - Quality is decided when you choose what to remember. A small, clean, deduplicated store retrieves far better than a large noisy one. - A few correct memories outperform many loosely related ones. Tune for relevance and rank tightly rather than maximizing how much you inject. - Store metadata and provide ways to view, edit, expire, and delete memories. This is essential for debugging, trust, and meeting privacy obligations. - Facts go stale and contradict each other. Build consolidation, supersession, and expiry early; retrofitting them onto a large polluted store is painful. ### FAQs **How is this different from semantic caching?** Semantic caching stores and replays whole answers to avoid recomputing similar requests. Long-term memory stores durable facts, preferences, and outcomes, then recomposes them into fresh reasoning for each new task. One reuses outputs; the other remembers state. **What should the agent actually remember?** Stable, reusable signal: user preferences, decisions and commitments, outcomes of prior tasks, and recurring procedures. Avoid transient chatter and anything you cannot justify retaining. Writing less but writing well is what makes later retrieval precise. **How do you handle PII and privacy?** Treat the store as governed data: enforce scope so memories never cross users or tenants, minimize what you persist, support consent and deletion, and set retention and access controls. Inspectability and an expiry policy are part of meeting these obligations. ### References - Packer et al. — MemGPT (2023) — https://arxiv.org/abs/2310.08560 - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents Related: semantic-caching, context-compression --- ## Orchestrator-Workers URL: https://santismm.com/en/patterns/orchestrator-workers | API: https://santismm.com/api/patterns/orchestrator-workers Category: orchestration | Updated: 2026-06-24 | Version: 1.1 Evidence: production | Confidence: low | Source: production_system, personal_experience, industry_observation An orchestrator LLM dynamically breaks a task into subtasks, delegates each to a worker LLM, and synthesizes the results. Unlike fixed parallelization, the orchestrator decides the subtasks at runtime — making it suited to complex tasks whose decomposition is not known in advance. ### Problem Some tasks are too complex for a single call and cannot be decomposed up front, because the needed subtasks depend on the input. ### When to use it Use orchestrator-workers when a task needs dynamic decomposition — the number and nature of subtasks vary by input — and a coordinating model can plan and integrate the work. ### Solution A lead (orchestrator) model analyzes the task, decides which subtasks are needed, and delegates each to a worker model (often specialized). It then collects and synthesizes the workers' outputs into a final result. It is the agentic generalization of parallelization: the decomposition is decided at runtime rather than hard-coded, which adds flexibility at the cost of more coordination and unpredictability. ### Components - Orchestrator (lead) model - Worker models - Delegation logic - Synthesizer - Shared state / tools ### Benefits - Handles complex tasks with dynamic decomposition. - Workers can be specialized per subtask. - Scales to varied inputs without hard-coded steps. ### Risks - Coordination overhead, latency and token cost. - Harder to predict and debug than fixed workflows. - The orchestrator can mis-plan or loop without limits. ### When not to use it - When the decomposition is known in advance — use chaining or fixed parallelization. - For simple tasks a single call handles. - When predictability and tight cost control are paramount. ### Technologies - LangGraph - CrewAI - OpenAI Agents SDK - Model Context Protocol (MCP) ### Examples - A coding task where the lead decides which files to change and delegates edits. - A research task split into sub-questions, each researched then synthesized. - A complex report assembled from dynamically chosen sections. ### Production evidence - Context: Single-operator, local-first OpenClaw deployment observed over 57 days (161 sessions / 2,776 turns), aggregated from the agent's own trajectory traces. - Scenario: Scheduled work runs in isolated, one-shot worker sessions forked from the parent, with capability scoping and child/depth limits. - Technology: Cron isolated-agent runtime, session forking (forkSessionFromParent), a subagent lane and registry, and per-agent child/depth limits. - Load: 57 cron-isolated worker sessions over the window (the explicit sessions.spawn tool was not exercised). - Results: 57 isolated worker sessions ran without cross-session interference; forked-session isolation was the dominant worker pattern, while the explicit spawn tool stayed unused in this window. Single-operator local-first deployment. ### KPIs - End-to-end task completion rate: Share of orchestrated jobs that finish correctly across all sub-tasks; the orchestrator owns the whole outcome. - Worker fan-out & cost: Number of worker calls per job and their combined token cost; orchestration can explode spend if decomposition is sloppy. - Critical-path latency: Wall-clock of the longest dependent chain, not the sum of workers — this bounds responsiveness. - Sub-task error rate: How often individual workers fail or return unusable results, driving retries and recovery. ### Observed failure modes - Bad decomposition: the orchestrator splits the task wrongly, so correct workers still produce a wrong whole. - Context loss between orchestrator and workers, causing inconsistent or contradictory partial results. - Cost blow-up from spawning too many workers or deep nesting without budget limits. - Single point of failure: if the orchestrator misjudges, the entire job fails despite healthy workers. ### Lessons learned - Invest in the decomposition logic — most failures trace back to how the work was split, not the workers. - Pass workers the minimum context they need, explicitly, to avoid drift and contradictions. - Set a budget and depth cap; orchestration without limits is where agent cost spirals. - Make the orchestrator's plan inspectable so failures can be traced to a specific sub-task. ### FAQs **How is this different from parallelization?** Parallelization uses a fixed, predefined split. Orchestrator-workers decides the subtasks dynamically at runtime, so it handles tasks whose shape varies by input. **Is this a multi-agent system?** Yes — it is a common multi-agent pattern. Use it only when a task genuinely benefits from dynamic, separable subtasks. **How do I keep it from running away?** Set budgets, step limits and stop conditions, and add observability so you can see and bound the orchestrator's planning. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents Related: routing, parallelization, evaluator-optimizer --- ## Parallelization URL: https://santismm.com/en/patterns/parallelization | API: https://santismm.com/api/patterns/parallelization Category: orchestration | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation Parallelization runs multiple LLM calls at the same time and aggregates the results. Two flavors: sectioning (split a task into independent subtasks run in parallel) and voting (run the same task several times to improve reliability or coverage). It cuts latency and can raise quality. ### Problem Running independent subtasks one after another wastes time, and a single sample of a hard task can be unreliable. ### When to use it Use parallelization when subtasks are independent (sectioning), or when multiple attempts at the same task improve confidence or coverage (voting). ### Solution Sectioning: split the work into independent pieces, run them concurrently, and combine the outputs. Voting: run the same prompt multiple times (or with variations) and aggregate by majority, union or a judge. Both reduce wall-clock time versus sequential execution; voting additionally trades extra cost for higher reliability on tasks where a single sample is risky. ### Components - Task splitter - Concurrent workers - Aggregator (merge / vote / judge) ### Benefits - Lower latency by running calls concurrently. - Voting improves reliability and coverage. - Each parallel call stays simple and focused. ### Risks - Voting multiplies token cost. - Aggregation logic can be tricky to get right. - Subtasks assumed independent may actually interact. ### When not to use it - When subtasks depend on each other's output — chain them. - When cost is tight and a single call suffices. - When results cannot be aggregated meaningfully. ### Technologies - LangGraph - Async runtimes - OpenAI Agents SDK - Map-reduce frameworks ### Examples - Summarizing many documents at once, then merging the summaries. - Running a safety check in parallel with the main response. - Sampling an answer several times and taking the majority. ### KPIs - Latency reduction vs. sequential: Wall-clock saved by running calls concurrently; the whole point of the pattern. - Aggregation quality: Whether merging the parallel outputs preserves correctness — the hard part is the join, not the fan-out. - Concurrency cost: Total tokens across all parallel branches; you trade money for speed, so watch the multiplier. - Rate-limit / throttle rate: How often parallel calls hit provider rate limits, which silently serializes or fails them. ### Observed failure modes - Aggregation errors: parallel results are correct individually but combined wrongly (double-counting, contradictions). - Rate limiting turns intended parallelism back into slow, serialized calls. - Cost surprise: N parallel branches cost N× even when only one result is used. - Partial failure handling: one branch fails and the aggregator either blocks or silently drops it. ### Lessons learned - Design the aggregation step first — combining results well is harder than splitting the work. - Respect provider rate limits with batching or backoff, or parallelism evaporates. - Only parallelize independent sub-tasks; dependencies force a sequence anyway. - Decide explicitly how partial failures are handled before they happen in production. ### FAQs **What is the difference between sectioning and voting?** Sectioning splits one task into different independent subtasks; voting runs the same task multiple times to aggregate for reliability. **Does voting always improve quality?** Often, on tasks where samples vary, but it multiplies cost. Reserve it for high-stakes steps where a single sample is risky. **How do I combine parallel results?** Depending on the case: concatenate sections, take a majority vote, union the findings, or use a judge model to synthesize. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents Related: prompt-chaining, orchestrator-workers, evaluator-optimizer --- ## Prompt Chaining URL: https://santismm.com/en/patterns/prompt-chaining | API: https://santismm.com/api/patterns/prompt-chaining Category: orchestration | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper Prompt chaining decomposes a task into a fixed sequence of LLM calls, where each step works on the output of the previous one. It trades a little latency for much higher accuracy and control, and is the simplest workflow pattern: use it whenever a task cleanly splits into ordered subtasks. ### Problem A single prompt asked to do several things at once produces lower-quality, harder-to-control output, and is difficult to debug when it goes wrong. ### When to use it Use prompt chaining when a task decomposes into a clear, ordered sequence of subtasks — for example outline, then draft, then edit — and each step benefits from the previous step's result. ### Solution Break the task into discrete steps and run one LLM call per step, passing each output to the next. Optionally add programmatic checks (gates) between steps to validate intermediate results before continuing. Because each call has one focused job, prompts are simpler, outputs are more reliable, and failures are localized to a specific step that you can inspect and fix. ### Components - Ordered steps - Per-step prompt - Inter-step gates / validation - State passed between steps ### Benefits - Higher accuracy by giving each call one focused job. - Easier to debug — failures localize to a step. - Validation gates can catch errors between steps. ### Risks - Higher total latency from sequential calls. - Errors can compound down the chain if not checked. - Too many steps add cost and brittleness. ### When not to use it - When the task is simple enough for a single call. - When subtasks are independent — parallelize instead. - When the path is unknown up front — use an agent loop. ### Technologies - LangGraph - OpenAI Agents SDK - Claude Agent SDK - Workflow engines ### Examples - Generate an outline, then write each section, then revise for tone. - Extract structured fields, then validate them, then summarize. - Translate a document, then check the translation against the source. ### KPIs - End-to-end success rate: Share of chains that produce a correct final result; errors compound across steps. - Per-step error rate: Failure rate at each link — a 95%-reliable step chained five times yields ~77% end to end. - Total latency & cost: Sum across every call in the chain; more steps mean more of both. - Recovery rate: How often a failed intermediate step is caught and corrected rather than silently propagated. ### Observed failure modes - Error propagation: a mistake early in the chain corrupts every downstream step. - Latency and cost accumulation as the chain grows longer. - Brittle hand-offs when one step's output format doesn't match the next step's expected input. - Lost context across steps, so later links forget constraints set earlier. ### Lessons learned - Validate or gate-check between steps so errors are caught before they propagate. - Keep chains as short as the task allows; every extra step multiplies failure probability. - Pin the output contract of each step so hand-offs don't break silently. - Use chaining for genuinely sequential work; parallelize independent steps instead. ### FAQs **How is prompt chaining different from an agent?** Prompt chaining follows a fixed, predefined sequence. An agent decides its own steps dynamically. Prefer chaining when the path is known in advance. **When should I add gates between steps?** Whenever an intermediate result must meet a condition before proceeding — it stops errors from propagating down the chain. **Does chaining increase cost?** Yes, modestly — more calls mean more tokens and latency — but the gain in reliability usually outweighs it for multi-part tasks. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents Related: routing, orchestrator-workers, evaluator-optimizer --- ## Recovery Strategy URL: https://santismm.com/en/patterns/recovery-strategy | API: https://santismm.com/api/patterns/recovery-strategy Category: reliability | Updated: 2026-06-24 | Version: 1.1 Evidence: production | Confidence: low | Source: production_system, personal_experience, industry_observation Give the agent an explicit plan for when things break. Detect failures by validating outputs and catching tool errors; then retry with adjustment, fall back to an alternative path, roll back partial actions, or escalate. Bound retries to avoid runaway loops and cost, make actions idempotent, and distinguish transient from permanent failures. The goal is graceful degradation instead of crashes or silently wrong results. ### Problem Agents fail constantly: tools time out, APIs return errors, models emit malformed output, plans hit dead-ends, and multi-step workflows leave partial side effects behind. Without an explicit recovery path, an agent either crashes on the first error or — worse — plows ahead on bad data and silently produces confidently wrong results. Naive retry loops make it worse, hammering a failing dependency, burning tokens, and spinning forever. The hard part is not catching one error; it is deciding what kind of failure it is and what response is safe. ### When to use it Use this pattern in any agent that calls external tools, runs multi-step plans, or takes consequential actions where partial completion is possible. It matters most for long-running or autonomous workflows that no human watches in real time, and for actions with side effects (payments, writes, emails) where a blind retry could duplicate work. It assumes you can validate outputs against some contract and that at least some operations can be made idempotent or compensated. It is less relevant for single-shot, read-only, low-stakes prompts. ### Solution Treat recovery as a first-class control loop layered around the agent's normal execution. Every tool call and model output passes through a validation gate: catch exceptions and timeouts, and check outputs against a schema or contract before trusting them. On failure, classify it. Transient failures (timeouts, rate limits, 5xx) get a bounded retry with exponential backoff and jitter, ideally against an idempotent operation so a duplicate request is harmless. Permanent failures (invalid arguments, auth errors, contract violations) skip retries and move straight to an alternative: a different tool, a simpler plan, or a fallback answer. When progress matters, checkpoint state so the agent can resume from the last good step rather than restarting. When a step has already produced side effects and cannot proceed, run compensating actions to roll back — cancel the order, delete the draft, reverse the charge. Wrap the whole loop in hard budgets: maximum attempts, maximum wall-clock time, and a cost ceiling, plus a circuit breaker that stops calling a dependency that keeps failing. When all recovery options are exhausted, escalate cleanly — surface the failure to a human or a supervising agent with enough context to act, rather than guessing. ### Components - Validation gate - Failure classifier - Bounded retry with backoff - Fallback router - Checkpoint store - Compensation handler ### Benefits - The agent produces a partial or fallback result and a clear status instead of crashing or returning confident garbage. - Hard attempt, time, and cost limits stop runaway retry loops from burning budget on a failing dependency. - Compensating actions and checkpoints keep external systems and task state coherent when a workflow stops midway. - When recovery fails, the agent hands off with enough context for a human or supervisor to act, rather than guessing. ### Risks - Aggressive retries against a struggling dependency add load and can turn a brief blip into a cascading outage. - Retrying a non-idempotent action can double-charge, double-send, or double-write if request keys are not used. - Over-eager fallbacks can hide systematic failures, so a broken tool looks healthy while quietly degrading every result. - Rollback logic is often incomplete or itself fails, leaving systems in an inconsistent state that is hard to detect. ### When not to use it - For low-stakes prompts with no side effects and no multi-step plan, a simple retry-or-fail is enough; full recovery machinery is overhead. - When a failure means the task is genuinely impossible (missing permission, deprecated API), retry and fallback only waste time — fail fast and escalate. - If a side effect is irreversible and cannot be made idempotent, do not auto-retry across it; require confirmation or human approval instead. ### Technologies - Retries with backoff - Circuit breakers - Checkpointing - Compensating actions ### Examples - A research agent's search tool returns a 503; the agent retries with backoff, succeeds on the third attempt, and continues without crashing the run. - A code agent generates JSON that fails schema validation; the validation gate rejects it and re-prompts with the error, instead of passing malformed data downstream. - A booking agent reserves a flight but the hotel step fails permanently; the compensation handler cancels the reservation and escalates rather than leaving a half-booked trip. ### Production evidence - Context: Single-operator, local-first OpenClaw deployment observed over 57 days (161 sessions / 2,776 turns), aggregated from the agent's own trajectory traces. - Scenario: Errors, aborts and timeouts during autonomous turns are absorbed by retry/backoff and a model-fallback chain so the agent keeps running. - Technology: retryAsync with exponential backoff and jitter, runWithModelFallback, abort propagation, and cron self-protection against refire loops. - Load: 2,942 terminal turns; 52 errors, 126 aborts, 120 timeouts and 31 prompt-errors observed. - Results: A 1.77% terminal-error rate across 2,942 turns; recovery primitives absorbed transient failures and the deployment held 98.8% session success. Single-operator local-first deployment. ### KPIs - Recovery success rate: Share of failures resolved automatically by retry or fallback without human help; a healthy value is high and stable, with no quiet downward drift. - Mean attempts per successful task: How many tries it takes to succeed; watch for creep, which signals a degrading dependency rather than genuine recovery. - Unbounded-loop / budget-breach rate: How often runs hit retry, time, or cost ceilings; this should be rare, and spikes mean limits or classification need tuning. - Compensation completeness: Fraction of failed multi-step workflows that end in a consistent state; the target is full rollback with no orphaned side effects. ### Observed failure modes - Missing or too-high attempt caps let the agent retry a permanent failure forever, burning cost and never making progress. - Treating a permanent error as transient wastes retries; treating a transient error as permanent gives up too early and triggers needless fallbacks. - A fallback path returns a plausible but wrong answer with no signal that recovery occurred, so downstream consumers trust bad output. - The agent fails after a write or external action but before compensation runs, leaving duplicate or dangling records. ### Lessons learned - The retry/fallback/escalate decision hinges on transient versus permanent; invest in clear classification before tuning backoff curves. - Idempotency keys turn a risky retry into a safe one; design for it up front rather than bolting on dedup later. - Hard caps on attempts, time, and cost are non-negotiable; an agent without them will eventually find a way to run forever. - Log every retry, fallback, and compensation so silent degradation surfaces as a metric instead of a surprise incident. ### FAQs **How is this different from just adding try/except and a retry loop?** Try/except handles one error; a recovery strategy decides what kind of failure it is and chooses among retry, fallback, rollback, and escalation under hard budgets. The control logic and idempotency, not the exception handling, are the substance. **How many retries should I allow?** Few — typically a small fixed cap with exponential backoff and jitter, plus separate time and cost ceilings. The exact number depends on the dependency, but the loop must always terminate, and permanent failures should not be retried at all. **What if an action cannot be undone or made idempotent?** Do not auto-retry across it. Checkpoint before the irreversible step, and on failure escalate to a human or supervising agent rather than guessing. Irreversibility is a signal to slow down, not to retry harder. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - Google SRE Book — Handling Overload & Cascading Failures — https://sre.google/sre-book/handling-overload/ Related: reflection, human-escalation, evaluator-optimizer --- ## Reflection URL: https://santismm.com/en/patterns/reflection | API: https://santismm.com/api/patterns/reflection Category: reliability | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper **Definition.** Reflection is a pattern in which a model reviews and critiques its own output against explicit criteria and then revises it, trading extra inference for higher quality. Reflection has a model critique its own output and then revise it, using the critique as feedback. It is a lightweight, single-model way to catch mistakes and improve quality on reasoning, coding and writing tasks — at the cost of extra calls. ### Problem Models often produce a flawed first answer they could improve if prompted to review their own work, but a single pass gives them no chance to. ### When to use it Use reflection when a self-review step measurably improves output and you want a simpler alternative to a two-model evaluator loop — common in reasoning and coding tasks. ### Solution After generating an answer, prompt the same model to critique it against the goal (and any tool feedback such as test results or errors), then to produce a revised answer informed by that critique. Repeat for a bounded number of iterations. Reflection works best when grounded in real signals — execution errors, test output, retrieved facts — rather than pure self-assessment, which can be overconfident. ### Components - Initial generation - Self-critique step - Grounding signal (errors / tests / facts) - Revision - Iteration budget ### Benefits - Improves quality with a single model — no second system. - Effective when grounded in tool or test feedback. - Simple to add to an existing call. ### Risks - Self-critique can be overconfident or miss its own errors. - Extra calls add latency and cost. - Without grounding, gains are limited. ### When not to use it - When you have an objective external check — use evaluator-optimizer. - When a single pass already meets the bar. - When latency budgets are very tight. ### Technologies - LangGraph - Agent frameworks - LLM-as-judge ### Examples - A coding agent reading test failures and fixing its own patch. - A reasoning task where the model rechecks its steps before answering. - A draft the model reviews for gaps before finalizing. ### Production evidence - Context: Tasks where output quality matters more than latency or cost — drafting, code generation, analysis — and where errors are detectable on review. - Scenario: After producing a first answer, the model (or a separate critic) evaluates it against concrete criteria and produces a revised version; the loop is capped at one or two passes. - Technology: A critique-then-revise prompt chain, ideally backed by external signals (tests, tools, a separate evaluator) for high-stakes work. - Load: Each reflection pass at least doubles calls, so it is applied selectively to the outputs that justify the overhead. - Results: Observed pattern: reflection lifts quality where the model can actually detect its own errors, but it can over-revise correct answers and at least doubles cost. Measure the quality lift against an eval set before trusting it, and prefer external signals when stakes are high. ### KPIs - Quality lift from reflection: Measured improvement in output quality with the reflection step versus without; if it's not measurable, the step isn't earning its cost. - Self-correction rate: Share of genuine errors the model catches and fixes on review — distinct from cosmetic edits. - Added latency & cost: Reflection at least doubles calls; track the overhead against the quality it buys. - Over-revision rate: How often reflection degrades an already-good answer by second-guessing it. ### Observed failure modes - Self-evaluation blind spots: a model often can't see its own errors, so reflection misses them. - Over-revision: the model 'fixes' a correct answer into a worse one. - Cost and latency double (or more) for marginal or no quality gain. - False confidence: the model asserts the output is now correct when it isn't. ### Lessons learned - Measure the lift; reflection is worth it only where it demonstrably improves quality. - Prefer external signals (tests, tools, a separate evaluator) over pure self-critique when stakes are high. - Cap reflection to one or two passes — returns diminish fast and cost compounds. - Give the reflection step concrete criteria, not a vague 'improve this'. ### FAQs **Reflection or evaluator-optimizer?** Reflection uses one model to self-critique (simpler); evaluator-optimizer uses a separate evaluator (sharper, less biased). Choose by how reliable self-assessment is for your task. **Does reflection always help?** It helps most when grounded in real feedback like test results or errors. Pure self-assessment can be overconfident and add little. **How many reflection rounds?** Keep it bounded — often one or two. Diminishing returns and rising cost make long loops rarely worth it. ### References - Shinn et al. — Reflexion: Language Agents with Verbal Reinforcement Learning (2023) — https://arxiv.org/abs/2303.11366 Related: evaluator-optimizer, prompt-chaining --- ## Routing URL: https://santismm.com/en/patterns/routing | API: https://santismm.com/api/patterns/routing Category: orchestration | Updated: 2026-06-24 | Version: 1.1 Evidence: production | Confidence: low | Source: production_system, personal_experience, industry_observation **Definition.** Routing is a pattern that classifies each incoming request and dispatches it to the most appropriate handler or model, so easy inputs use cheap paths and hard inputs use capable ones. Routing classifies an input and directs it to the most appropriate specialized handler, prompt or model. It improves quality by letting each path be optimized for its case, and controls cost by sending easy requests to cheap models and hard ones to capable models. ### Problem A single prompt or model handling every kind of input does each one worse, and using one expensive model for everything wastes money on easy requests. ### When to use it Use routing when inputs fall into distinct categories that benefit from different handling — different prompts, tools, models or workflows — and the categories can be classified reliably. ### Solution A lightweight classifier (an LLM call or a model) labels the input, then a router sends it to the matching downstream handler. Each handler is specialized and optimized for its category. Routing also enables cost-performance tiering: route simple queries to a fast, cheap model and complex ones to a stronger reasoning model, paying for capability only when it is needed. ### Components - Classifier - Routing logic - Specialized handlers - Fallback / default route ### Benefits - Each path is optimized for its case, raising quality. - Cost control by tiering models to difficulty. - Separation of concerns keeps each handler simple. ### Risks - Misclassification sends inputs down the wrong path. - The classifier adds a step and some latency. - Category drift over time degrades routing accuracy. ### When not to use it - When inputs are homogeneous — one handler suffices. - When categories cannot be classified reliably. - When the added classification step is not worth the gain. ### Technologies - Classifier models - LangGraph - Model routers - Rules engines ### Examples - Routing support tickets to billing, technical or sales handlers. - Sending simple questions to a small model and hard ones to a reasoning model. - Directing different document types to type-specific extractors. ### Production evidence - Context: Single-operator, local-first OpenClaw deployment observed over 57 days (161 sessions / 2,776 turns), aggregated from the agent's own trajectory traces. - Scenario: Inbound channel messages and autonomous wake-ups are routed to one agent through distinct entrypoints, with channel/peer/role bindings selecting the session. - Technology: Binding + route registry (resolveAgentRoute), per-channel session keys, and a resolved-route cache. - Load: 3 channels (Telegram, web chat, WhatsApp) and 4 trigger kinds (user, heartbeat, cron, memory). - Results: Routing held across all three channels and four trigger types with no misroute surfacing as a failure (98.8% session success overall). Single-operator local-first deployment — a working reference, not a scale benchmark. ### KPIs - Routing accuracy: Share of inputs sent to the correct handler/model; the single metric that defines the pattern's value. - Cost savings vs. always-best-model: Money saved by routing easy inputs to cheaper models instead of the top one for everything. - Misroute cost: The downstream damage of wrong routes — a misroute can cost far more than the savings it chased. - Router latency overhead: Time the routing decision itself adds before any real work begins. ### Observed failure modes - Misclassification: the router sends an input to the wrong model or path, degrading the answer. - Ambiguous inputs that don't fit any route cleanly and get forced into a poor one. - Router becomes a bottleneck or single point of failure for every request. - Drift: input distribution shifts over time and the router's categories go stale. ### Lessons learned - Optimize for the cost of a misroute, not just routing accuracy — some wrong routes are far costlier than others. - Add a default / fallback route for inputs that match nothing well. - Keep the router cheap and fast; if it costs as much as the work, it defeats the purpose. - Monitor input drift and re-tune routes as the distribution changes. ### FAQs **What classifies the input?** Usually a lightweight LLM call or a dedicated classifier model; for clear-cut cases, deterministic rules can route without a model. **How does routing save cost?** By tiering: easy requests go to cheap, fast models and only hard ones reach expensive reasoning models, so you pay for capability only when needed. **What if the classifier is wrong?** Provide a sensible default route and monitor misroutes; a fallback handler and good observability limit the impact of misclassification. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents Related: prompt-chaining, orchestrator-workers, parallelization --- ## Semantic Caching URL: https://santismm.com/en/patterns/semantic-caching | API: https://santismm.com/api/patterns/semantic-caching Category: cost | Updated: 2026-06-24 | Version: 1.1 Evidence: production | Confidence: low | Source: production_system, personal_experience, industry_observation Semantic caching stores past model responses and reuses them when a new request is semantically similar to a previous one — matching by meaning via embeddings, not exact text. It cuts cost and latency for repetitive or near-duplicate queries common in production. ### Problem Many production queries are paraphrases of ones already answered, so re-running the full model on each wastes cost and latency. ### When to use it Use semantic caching when traffic contains many similar or repeated questions and answers are stable enough to reuse — FAQs, support, documentation assistants. ### Solution Embed each incoming request and search a cache of prior request embeddings. If a sufficiently similar entry exists (above a similarity threshold), return its stored response; otherwise call the model and store the new pair. Tune the similarity threshold carefully: too loose returns wrong answers for subtly different questions; too strict misses valid hits. Add TTLs and invalidation so cached answers do not go stale. ### Components - Embedding of the request - Vector cache - Similarity threshold - TTL / invalidation - Fallback to model ### Benefits - Lower cost by avoiding repeat model calls. - Lower latency on cache hits. - More consistent answers to similar questions. ### Risks - A loose threshold serves wrong cached answers. - Stale cache without TTL or invalidation. - Personalized or time-sensitive answers cache poorly. ### When not to use it - When most queries are unique. - When answers depend on fresh, user- or time-specific data. - When even small mismatches are unacceptable. ### Technologies - Embedding models - Vector databases - GPTCache - Redis / KV stores ### Examples - Reusing the answer to 'how do I reset my password' across its many phrasings. - Caching common documentation questions in a support assistant. - Short-circuiting repeated identical analytics questions. ### Production evidence - Context: Single-operator, local-first OpenClaw deployment observed over 57 days (161 sessions / 2,776 turns), aggregated from the agent's own trajectory traces. - Scenario: Prompt and context caching is engineered with cache_control markers plus a workspace file cache and a route cache, so repeated structure is served from cache. - Technology: Anthropic cache_control injection on system and messages, OpenRouter passthrough, workspace file cache, route cache, and cache_read/write cost accounting. - Load: 28.1M total tokens over 57 days, of which ~19.6M were served as cache-read. - Results: About 70% of tokens were served from cache, holding blended cost to $15.21 per 1M tokens ($41 of cache-read versus $374 of fresh input). Single-operator local-first deployment — the ratio reflects this workload's repetition; measure your own. ### KPIs - Cache hit rate: Share of requests served from cache; the lever for both cost and latency savings. - False-hit rate: How often a semantically 'similar' hit returns a wrong or stale answer — the central risk of caching by meaning. - Cost & latency saved per hit: Tokens and time avoided on cache hits, the upside you're trading the false-hit risk for. - Similarity threshold calibration: Whether the match threshold balances hit rate against false hits; too loose hurts quality, too strict kills savings. ### Observed failure modes - False hits: two queries are similar in embedding space but need different answers, so the cache returns a wrong one. - Staleness: cached answers go out of date while the underlying facts change. - Threshold mis-tuning: too loose returns wrong answers, too strict yields almost no hits. - Cache poisoning: a bad answer gets cached and then served repeatedly. ### Lessons learned - Tune the similarity threshold against real traffic; it is the make-or-break parameter. - Never cache where freshness or correctness is critical without an invalidation strategy. - Validate or sample cache hits to catch false matches before users do. - Scope caches narrowly (per tenant, per context) to avoid leaking the wrong answer across users. ### FAQs **How is this different from a normal cache?** A normal cache matches exact keys; a semantic cache matches by meaning using embeddings, so paraphrased questions still hit. **What is the main risk?** A too-loose similarity threshold returns a cached answer for a question that is actually different. Tune the threshold and validate on real traffic. **How do I avoid stale answers?** Set TTLs and invalidate entries when the underlying data changes; avoid caching personalized or time-sensitive responses. ### References - OpenAI — Vector embeddings guide — https://platform.openai.com/docs/guides/embeddings Related: routing, prompt-chaining --- ## Supervisor Agent URL: https://santismm.com/en/patterns/supervisor-agent | API: https://santismm.com/api/patterns/supervisor-agent Category: orchestration | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper A supervisor agent is a persistent coordinator that manages a team of specialized sub-agents. It reads the conversation state, decides which specialist should act next, routes messages to it, and integrates returned results toward the goal. Unlike a one-shot decomposer, the supervisor stays in the loop across many turns, delegating by capability and re-planning until the task is done or handed back to the user. ### Problem A single agent given many tools, instructions, and domains becomes unfocused: its prompt bloats, tool selection degrades, and it confuses unrelated concerns. Real workflows need different expertise at different steps (research, coding, billing, compliance), but no single flat agent reliably picks the right capability at the right moment or keeps long multi-step interactions coherent. ### When to use it Use a supervisor when work spans several distinct, reusable specialist capabilities that must collaborate over a multi-turn conversation or loop, when routing decisions depend on evolving state rather than a fixed plan, and when you need a clear, central place to enforce policy, manage handoffs, and observe which agent did what. It fits heterogeneous teams of agents more than uniform parallel workers. ### Solution The supervisor owns the control loop and the shared conversation state. On each turn it inspects the latest messages and goal, then decides whether to answer directly, delegate to a named specialist, or finish. Delegation is by capability: each sub-agent has a declared scope (for example a code agent, a data agent, a knowledge agent), and the supervisor routes the relevant slice of context to the chosen one. The specialist runs its own focused tool loop and returns a result or a request for clarification, which the supervisor records before deciding the next step. Control returns to the supervisor after every specialist turn, so it remains the single decision point rather than letting agents call each other freely. The supervisor integrates partial results, resolves conflicts between specialists, decides when a goal is satisfied, and decides when to hand back to the user. Guardrails such as step budgets, allowed-transition rules, and explicit termination conditions keep the loop from cycling. Structured handoff messages and a shared trace make every delegation auditable, so teams can see who was asked to do what and why. ### Components - Supervisor (router/planner) - Specialist sub-agents with declared scopes - Shared conversation/state store - Handoff protocol and message schema - Step budget and termination guard - Trace and per-agent observability ### Benefits - Focused specialists with smaller, cleaner prompts - Centralized routing and policy enforcement - Modular agents that can evolve independently - Clear audit trail of who did what ### Risks - Infinite or ping-pong handoff loops - Coordination overhead inflates latency and cost - Supervisor becomes a routing bottleneck - Context loss across handoffs degrades quality ### When not to use it - Single capability handles the whole task - Fixed parallel fan-out fits better (orchestrator-workers) - Latency or cost budgets forbid extra hops ### Technologies - LangGraph (supervisor) - OpenAI Agents SDK - Multi-agent frameworks - Message routing ### Examples - Customer support routing across billing, technical, and account specialists - Software task split among coding, testing, and documentation agents - Research assistant delegating to search, analysis, and writing agents ### KPIs - Task success / goal-completion rate: Share of sessions reaching the intended outcome without human rescue; the headline quality signal for the supervisor team. - Handoffs per resolved task: Average delegations to completion; watch for upward drift signaling indecision or routing thrash, not richer work. - Coordination overhead: Extra tokens, calls, and latency attributable to the supervisor versus a single agent; good means routing earns its cost. - Routing accuracy: Fraction of delegations sent to the correct specialist on first try, judged against labeled cases. ### Observed failure modes - Two agents hand work back and forth without progress until a budget cuts the loop - Supervisor mis-routes to the wrong specialist and never recovers the thread - Critical context is dropped in the handoff, so the specialist solves the wrong problem - Specialists' partial results conflict and the supervisor merges them incoherently ### Lessons learned - Enforce hard step budgets and explicit termination so loops always end - Make handoffs structured, with intent and scope, not raw message dumps - Keep specialist scopes narrow and non-overlapping to reduce routing ambiguity - Instrument every delegation; you cannot debug a multi-agent loop you cannot see ### FAQs **How is this different from orchestrator-workers?** Orchestrator-workers decomposes one task into parallel, often homogeneous worker calls and merges them. A supervisor is a persistent coordinator over heterogeneous specialists across a multi-turn loop, re-deciding routing as state evolves rather than executing a fixed plan. **How do I prevent infinite handoff loops?** Route control back to the supervisor after each specialist turn, forbid free peer-to-peer calls, set a step or token budget, define allowed transitions, and add explicit termination conditions so the loop cannot cycle indefinitely. **When should a specialist hand back to the supervisor?** Whenever it finishes its scoped task, needs a capability it does not own, hits ambiguity needing a decision, or detects it is the wrong agent for the request. The supervisor then integrates and picks the next step. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - LangGraph — Multi-agent systems — https://langchain-ai.github.io/langgraph/concepts/multi_agent/ Related: orchestrator-workers, routing, goal-decomposition --- ## Task Prioritization URL: https://santismm.com/en/patterns/task-prioritization | API: https://santismm.com/api/patterns/task-prioritization Category: orchestration | Updated: 2026-06-24 | Version: 1.1 Evidence: production | Confidence: low | Source: production_system, personal_experience, industry_observation Order an agent's candidate tasks by value, urgency, dependencies, and cost instead of processing them first-in-first-out. A scoring function and a priority queue decide what runs next, so limited compute, budget, and time go to the work that matters most. Re-score as state changes, and bound the queue so it cannot grow without limit. ### Problem An agent that decomposes a goal often ends up with many candidate tasks at once: searches to run, files to read, tools to call, sub-goals to pursue. Processing them in arrival order treats a trivial cleanup step as equal to a blocking, deadline-bound task. Important work waits behind cheap noise, dependencies are violated, and budget is spent on tasks that no longer matter once the situation has changed. ### When to use it Use this when an agent or orchestrator holds a backlog of independent or loosely coupled tasks and cannot run them all immediately because of compute, rate-limit, cost, or wall-clock constraints. It fits planner and supervisor architectures where one component chooses what executes next. It assumes you can attach signals — impact, deadline, dependency, cost — to each task, and that priorities may shift as new observations arrive. ### Solution Attach explicit signals to every task: expected impact toward the goal, urgency or deadline, dependency relationships (what must finish first), and estimated cost in tokens, money, or latency. Combine these into a single score with a transparent, auditable function rather than an opaque model judgment. Feed scored tasks into a priority queue so the highest-value ready task runs next. Always respect dependencies first: a task whose prerequisites are unmet is not 'ready' regardless of its score, which keeps the ordering correct and prevents wasted retries. Make prioritization dynamic. After each step, re-score affected tasks because new results change impact, deadlines approach, and some tasks become obsolete and can be dropped. Protect against starvation by aging — gradually raising the priority of long-waiting tasks — or by reserving capacity for lower tiers. Bound the backlog with an explicit cap and an admission policy: when the queue is full, reject, merge, or evict the weakest tasks instead of letting it grow without limit. Keep the scoring weights configurable and log why each task was chosen so behavior stays explainable. ### Components - Signal extractor - Scoring function - Priority queue - Dependency resolver - Re-prioritization loop - Admission and aging controller ### Benefits - Limited compute, budget, and time are spent on high-value, time-critical work instead of whatever arrived first. - Respecting prerequisites avoids wasted retries and rework caused by running tasks before their inputs exist. - Re-scoring lets the agent abandon now-irrelevant tasks and promote newly urgent ones as the situation evolves. - Cost-aware scoring and a bounded queue keep token and latency budgets under control rather than open-ended. ### Risks - A wrong weighting or bad cost estimate can systematically starve important work or chase low-value tasks; the formula needs review and calibration. - Without aging or reserved capacity, low-priority tasks may never run, leaving necessary cleanup or background work permanently undone. - Over-eager re-scoring can cause the agent to switch focus constantly, paying context-switch cost and never finishing anything. - If decomposition adds tasks faster than they are completed, an uncapped backlog inflates memory, cost, and planning latency. ### When not to use it - When there are only a handful of similar tasks, FIFO or simple parallelism is simpler and the scoring overhead is not worth it. - If tasks must run in a fixed sequence dictated by the domain, a static workflow or DAG is clearer than a dynamic priority queue. - When you can run everything immediately within budget and limits, there is nothing to prioritize and ordering adds needless complexity. ### Technologies - Task queues - Planner agents - Scheduling / priority queues - Cost-aware routing ### Examples - An agent gathering evidence prioritizes the searches most likely to resolve open questions and skips redundant queries once a claim is confirmed. - An operations agent orders remediation steps by blast radius and deadline, handling the customer-facing outage before low-impact warnings. - A pipeline agent schedules high-value or near-deadline documents first and defers cheap bulk items, while aging prevents the bulk queue from stalling forever. ### Production evidence - Context: Single-operator, local-first OpenClaw deployment observed over 57 days (161 sessions / 2,776 turns), aggregated from the agent's own trajectory traces. - Scenario: An autonomous agent is woken on a schedule (heartbeat + cron), with a top-of-hour stagger and a heartbeat cooldown providing back-pressure so jobs don't dogpile. - Technology: CronService scheduler, heartbeat-runner, a 5-minute top-of-hour stagger, heartbeat-cooldown defer logic, and a low/normal/high priority field on heartbeat outcomes. - Load: 2,801 heartbeat turns + 57 cron turns + 106 user turns over 57 days (heartbeat ~94% of triggers). - Results: The scheduler sustained roughly fifty autonomous wake-ups per day with stagger and cooldown preventing dogpiling; no schedule-driven failure mode surfaced. Single-operator local-first deployment. ### KPIs - Weighted task value completed per unit cost: Captures whether effort lands on high-impact work; good looks like more goal-relevant value delivered per token or dollar than a FIFO baseline. - Deadline / SLA adherence on time-critical tasks: Shows urgency signals are working; good looks like urgent tasks finishing before their deadline most of the time. - Starvation indicator (max and tail wait time for low-priority tasks): Reveals whether aging is effective; good looks like bounded worst-case waits with no task stuck indefinitely. - Queue depth vs. cap and admission/eviction rate: Confirms the backlog stays bounded; good looks like depth held under the cap with eviction reserved for genuinely low-value tasks. ### Observed failure modes - A high-priority task waits on a low-priority prerequisite that never gets scheduled; the resolver must propagate urgency to blockers. - Priorities computed once and never refreshed drive decisions on outdated impact or deadline information, so re-scoring must be triggered on relevant state changes. - Underestimating a task's cost lets it monopolize the budget; estimates need feedback from actual measured consumption. - An aggressive admission policy drops a task that later turns out to be required, forcing expensive rediscovery; eviction should prefer truly redundant items. ### Lessons learned - An auditable, configurable formula is easier to debug and tune than an opaque model judgment about what to do next. - Treat prerequisite completion as a separate readiness check so a high score never lets a task jump ahead of its inputs. - Add aging or reserved capacity from the start; low-priority background work that never runs becomes a silent correctness gap. - A hard cap with a clear admission policy is the simplest defense against runaway decomposition inflating cost and latency. ### FAQs **How is this different from goal decomposition?** Decomposition produces the tasks; prioritization decides the order in which the resulting tasks are executed. They are complementary: decomposition fills the backlog, prioritization drains it sensibly. **Should the LLM itself score priorities?** It can propose signals like estimated impact, but combine them with a transparent, auditable function. A deterministic formula over named signals is easier to calibrate, log, and trust than a single opaque ranking call. **How do I stop low-priority tasks from never running?** Use aging to gradually raise the priority of long-waiting tasks, or reserve a fraction of capacity for lower tiers, and monitor tail wait time to confirm nothing is starved. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - Yao et al. — ReAct (2022) — https://arxiv.org/abs/2210.03629 Related: goal-decomposition, supervisor-agent ================================================================= # ENTERPRISE AI REFERENCE ARCHITECTURES ================================================================= ## ARCH-001 — Customer Service Agent URL: https://santismm.com/en/architectures/customer-service-agent | API: https://santismm.com/api/architectures/customer-service-agent Category: customer-experience | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: medium | Source: industry_observation, paper Patterns used: routing, human-approval-gate, semantic-caching, reflection | Builds on: ai-agent, tool-use, enterprise-rag, guardrails, human-in-the-loop, ai-observability **Definition.** The customer service agent architecture is a grounded, tool-using conversational agent that resolves customer requests autonomously within guardrails, escalating to humans by risk and confidence, with full tracing for evaluation. A reference architecture for an enterprise customer-service agent that resolves common requests end to end — answering from a grounded knowledge base, acting in the CRM and ticketing systems through tools, and escalating to a human when confidence is low or the action is high-impact. It pairs retrieval for grounding with risk-based human approval for safety, and is observable so every conversation can be evaluated and improved. ### Key concepts - Grounding: answers come from retrieved, citable knowledge, not the model's memory. - Tool use: the agent reads and writes to CRM/ticketing systems through well-described tools. - Risk-based escalation: low-confidence or high-impact actions go to a human gate. - Observability: every turn is traced so the system can be evaluated and improved. ### Architecture At the core is an orchestration loop that classifies the incoming request, retrieves relevant knowledge, decides whether it can answer or must act, and either responds, calls a tool, or escalates. Routing sends simple FAQs down a cheap retrieval-and-answer path and complex or sensitive cases down a richer, more careful path. Grounding is non-negotiable: the agent answers from a retrieval layer over the help center and policy docs, and cites its sources. When the request requires an action — issuing a refund, changing an order, closing a ticket — the agent prepares the action and routes high-impact ones through a human approval gate before execution. Cross-cutting layers make it safe and improvable: guardrails redact PII and block out-of-policy responses, a semantic cache absorbs repeated questions to cut cost and latency, and an observability layer traces every turn so conversations can be scored against an evaluation set. ### Request flow - 1. Intake: the user message arrives; PII is detected and redacted for logging. - 2. Route: classify intent and risk — FAQ, account action, or escalation candidate. - 3. Retrieve: pull grounding passages from the knowledge base (cache-checked first). - 4. Decide: answer from grounding, call a CRM/ticketing tool, or escalate. - 5. Gate: high-impact actions pause for human approval; low-impact ones execute. - 6. Respond: reply with citations; log the trace and outcome for evaluation. ### Components - Intent & risk router - Retrieval layer (RAG) with citations - CRM / ticketing tools - Human approval gate - Guardrails & PII redaction - Semantic cache - Observability & evaluation ### Reference scenario - Context: An illustrative B2C support desk handling order, billing and account questions across chat and email. - Scenario: Tier-1 requests (order status, password reset, policy questions) are resolved by the agent; refunds and account changes are drafted by the agent and approved by a human; anything ambiguous is escalated with full context. - Technology: Orchestration loop, RAG over the help center, function-calling tools into the CRM, a risk-based approval gate, and conversation tracing. - Load: Bursty, business-hours-heavy traffic with a long tail of rare intents; a small set of FAQs dominates volume, which the semantic cache absorbs. - Results: Reference target: most Tier-1 volume deflected with grounded, cited answers; high-impact actions kept behind a human gate; cost concentrated on the rare, complex cases rather than the repetitive ones. Numbers depend on your traffic mix and should be measured, not assumed. ### Benefits - Resolves common requests end to end while keeping risky actions human-gated. - Grounded, cited answers reduce hallucination and build customer trust. - Semantic caching and routing concentrate spend on the cases that need it. - Full tracing makes quality measurable and regressions catchable. ### Risks - Ungrounded answers if retrieval quality is poor. - Over-automation of actions that should stay human-gated. - PII leakage if guardrails are incomplete. - Approval bottlenecks if too many actions are gated. ### KPIs - Containment / deflection rate: Share of conversations resolved without a human; the headline value metric — but only meaningful alongside CSAT. - Grounded-answer accuracy: How often answers are correct and supported by a citation, measured against an eval set. - Escalation rate & quality: Share escalated to humans and whether those escalations were warranted; too high wastes the automation, too low risks bad outcomes. - Cost per resolved conversation: Total tokens, tools and cache effect per resolution; routing and caching should keep this low on the common path. - CSAT / resolution time: Customer satisfaction and time-to-resolution; guards against optimizing deflection at the expense of experience. ### Cost & scaling - Volume scales with the stateless orchestration loop; the vector store and tool backends are the real capacity limits. - The semantic cache flattens cost as repeat questions grow, so unit cost falls with scale on the common path. - Human approval is the bottleneck that does not scale linearly — keep the gated set small and triaged. - Cost is dominated by the rare, complex conversations, not the cached FAQ majority. ### Observed failure modes - Retrieval misses or returns stale policy, so the agent answers confidently but wrongly. - Tool errors (CRM timeouts, schema drift) leave actions half-applied without recovery. - Escalation overload when the router sends too much to humans, defeating the automation. - Cache false hits return a previous customer's context or an out-of-date answer. ### Lessons learned - Ground first: invest in retrieval quality before expanding autonomy — most wrong answers are retrieval failures. - Gate by risk, not by default; reserve human approval for irreversible or regulated actions. - Scope the cache per customer/context and validate hits, or it leaks the wrong answer. - Instrument from day one; you cannot improve what you cannot trace. ### Technologies - LangGraph / orchestration - RAG over a help-center knowledge base - CRM & ticketing tools (function calling) - Vector store - Guardrails / PII redaction - Observability (LangSmith / Langfuse) ### Examples - An order-status question answered instantly from the cache with a citation. - A refund the agent drafts and a human approves before it is issued. - An ambiguous billing dispute escalated to an agent with the full conversation context attached. ### FAQs **How is this different from a chatbot?** A chatbot answers; this architecture also acts — it uses tools to read and write enterprise systems — and it grounds answers in retrieved knowledge, escalating by risk rather than following fixed scripts. **Why keep a human in the loop at all?** Because some actions are irreversible or regulated. A risk-based approval gate keeps accountability with a person for high-impact steps while automating the safe majority. **What makes it reliable?** Grounding in retrieval, guardrails on inputs and outputs, and observability that lets you evaluate every conversation and catch regressions before they ship. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework - Santiago Santa María — The Stopwatch and the Exam — https://articles.santismm.com/the-stopwatch-and-the-exam/ Related: enterprise-knowledge-assistant --- ## ARCH-002 — Enterprise Knowledge Assistant URL: https://santismm.com/en/architectures/enterprise-knowledge-assistant | API: https://santismm.com/api/architectures/enterprise-knowledge-assistant Category: knowledge | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper Patterns used: routing, semantic-caching, evaluator-optimizer, prompt-chaining | Builds on: enterprise-rag, embeddings, context-engineering, guardrails, agentic-evaluation, ai-governance **Definition.** The enterprise knowledge assistant architecture is a permission-aware RAG system that answers employee questions from internal documents with citations, scoped to each user's access rights and continuously evaluated for quality. A reference architecture for an internal knowledge assistant that answers employee questions from the company's own documents — wikis, policies, tickets, code — with citations and respecting each user's access permissions. It combines hybrid retrieval and reranking for grounding, permission-aware filtering for security, and an evaluation harness so answer quality is measured rather than assumed. The hard parts are not the model; they are retrieval quality, access control and evaluation. ### Key concepts - Permission-aware retrieval: a user only ever retrieves documents they are allowed to see. - Hybrid search + reranking: combine keyword and vector search, then rerank for precision. - Citations: every answer links back to its source passages for verification. - Evaluation: answer quality is scored against a curated set, continuously. ### Architecture Content from many internal sources is ingested, chunked and embedded into a vector store, with each chunk tagged by its source document's access-control metadata. At query time the assistant routes the question, runs hybrid retrieval (keyword + vector) filtered to the user's permissions, reranks the candidates, and synthesizes a cited answer from the top passages. Security is structural, not bolted on: the access-control filter is applied during retrieval so the model never even sees documents the user cannot access. A semantic cache serves repeated questions cheaply, and guardrails keep answers within policy and flag low-confidence cases. Quality is governed by measurement: an evaluation harness scores answers for groundedness, correctness and citation accuracy against a curated set, and an optional evaluator-optimizer loop revises weak answers before they reach the user. Observability traces every query so failures can be diagnosed and fed back into the evals. ### Request flow - 1. Ingest (offline): chunk and embed documents; tag each chunk with access-control metadata. - 2. Route: classify the question and pick the retrieval strategy. - 3. Retrieve: hybrid search filtered to the user's permissions (cache-checked first). - 4. Rerank: reorder candidates for precision; keep the top passages. - 5. Synthesize: generate a cited answer; optionally revise it via an evaluator loop. - 6. Return & log: deliver answer with citations; trace and score for evaluation. ### Components - Ingestion & chunking pipeline - Embeddings + vector store - Permission-aware retrieval filter - Hybrid search & reranker - Answer synthesis with citations - Semantic cache - Evaluation harness & observability ### Reference scenario - Context: An illustrative internal assistant over a company's wiki, HR and IT policies, and engineering docs. - Scenario: Employees ask natural-language questions ('how do I expense travel?', 'what's our on-call policy?'); the assistant answers with citations, never surfacing documents the asker cannot access, and says 'I don't know' rather than guessing when retrieval is weak. - Technology: Ingestion pipeline, embeddings + vector store with ACL metadata, hybrid retrieval and reranking, an evaluation harness, and query tracing. - Load: Steady internal traffic with strong query overlap (a few policies drive most questions), so the cache hit rate is high and embeddings dominate the offline cost. - Results: Reference target: grounded, cited answers with no access-control leaks, and a measurable groundedness score that improves as retrieval is tuned. Treat all figures as things to measure on your corpus, not guarantees. ### Benefits - Turns scattered internal knowledge into instant, cited answers. - Permission-aware retrieval prevents access-control leaks by construction. - Citations make answers verifiable and build user trust. - An evaluation harness makes quality measurable and improvements demonstrable. ### Risks - Access-control leaks if permissions are not enforced at retrieval time. - Stale answers when the document corpus changes faster than re-indexing. - Confident hallucination when retrieval is weak and the model fills the gap. - Poor chunking that fragments meaning and degrades retrieval. ### KPIs - Groundedness: Share of answers fully supported by the cited passages; the core quality metric for a RAG assistant. - Retrieval recall@k: How often the right passage is in the top-k retrieved; most answer errors trace back to this. - Access-control leak rate: Any answer surfacing a document the user couldn't access — the metric that must stay at zero. - Cache hit rate & cost per query: Repeat-question coverage and unit cost; high overlap should make most queries cheap. - Abstention quality: How often the assistant correctly says 'I don't know' instead of hallucinating on weak retrieval. ### Cost & scaling - Offline embedding and indexing dominate ingestion cost and grow with corpus size and update frequency. - Query-time cost is mostly retrieval + generation; reranking adds latency you trade for precision. - The cache flattens cost as query overlap rises, so unit cost falls with adoption. - Re-indexing cadence is the real scaling tension: fresher answers cost more compute. ### Observed failure modes - Permission bypass: a chunk inherits the wrong ACL and surfaces in a user's results. - Retrieval gaps: the right document exists but chunking or embeddings miss it. - Staleness: an answer cites a superseded policy because re-indexing lagged. - Citation drift: the cited passage doesn't actually support the generated claim. ### Lessons learned - Enforce access control inside retrieval, not after generation — filtering the prompt is too late. - Most quality gains come from retrieval (chunking, hybrid search, reranking), not from a bigger model. - Make 'I don't know' a first-class answer; a wrong confident answer is worse than an abstention. - Stand up evaluation before scaling; without it, every change is a guess. ### Technologies - RAG (retrieval-augmented generation) - Embeddings + vector store - Hybrid search & reranking - Document-level access control - Evaluation harness - Observability (LangSmith / Langfuse) ### Examples - An employee asking the travel-expense policy and getting a cited, up-to-date answer. - A question about a restricted project correctly returning nothing for an unauthorized user. - A weak-retrieval query answered with 'I don't have a confident source for that' instead of a guess. ### FAQs **Isn't this just RAG?** RAG is the core, but the architecture is defined by what makes it enterprise-safe: permission-aware retrieval, citations, an evaluation harness and observability. Those are the parts that decide whether it can be trusted. **Why enforce permissions during retrieval?** So the model never sees documents the user can't access. Filtering after generation is too late — the content could already have leaked into the answer. **How do you keep answers from hallucinating?** Ground every answer in retrieved passages with citations, measure groundedness against an eval set, and let the assistant abstain when retrieval is weak rather than fill the gap. ### References - Lewis et al. — Retrieval-Augmented Generation (2020) — https://arxiv.org/abs/2005.11401 - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework Related: customer-service-agent --- ## ARCH-003 — Sales Copilot URL: https://santismm.com/en/architectures/sales-copilot | API: https://santismm.com/api/architectures/sales-copilot Category: sales | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: medium | Source: industry_observation, paper Patterns used: routing, human-approval-gate, reflection, semantic-caching | Builds on: ai-agent, tool-use, enterprise-rag, context-engineering, guardrails, ai-observability **Definition.** The sales copilot architecture is an agentic assistant that grounds account research, outreach drafting, and activity logging in CRM and product data while keeping a rep approval gate before anything reaches a customer. A sales copilot is an agent that assists reps end to end: it researches accounts from CRM and product data, drafts personalized outreach and follow-ups, logs activity, surfaces next-best-actions, and prepares meeting briefs. Everything is grounded in the company's CRM, deal, and product knowledge to avoid hallucinated claims about features or pricing. The rep stays in control: a human-approval gate sits in front of any outbound action, so the agent drafts but never sends to a customer on its own. Success is measured honestly through rep productivity and pipeline impact, not vanity activity counts. ### Key concepts - Grounding in CRM, deal, and product data so the agent reasons over the rep's real pipeline instead of generic assumptions. - A strict separation between drafting and sending, with a human-approval gate on every customer-facing action. - Tool use via function calling to read and write CRM records, search product knowledge, and schedule meetings. - Honest measurement of rep productivity and pipeline outcomes rather than raw volume of emails or logged tasks. ### Architecture At the core sits an orchestration loop that interprets the rep's intent, plans a short sequence of tool calls, and assembles context. A router classifies each request — research an account, draft an email, log a call, prepare a brief, or suggest a next-best-action — and selects the relevant tools and retrieval scope. Keeping the loop bounded and observable matters more than maximal autonomy: every step is logged so the team can inspect what data was read, what was drafted, and why. Grounding is supplied by RAG over product and deal data plus direct CRM function calls. Product specs, pricing rules, security documentation, and battlecards live in a retrieval index; live account state — open opportunities, contacts, recent activity, stage — comes from CRM reads. The agent must cite or attach the source for any factual claim about a product or price, and guardrails reject outputs that assert unverifiable specifics. This is the main defense against fabricated features or numbers leaking into customer communication. Outbound actions are gated. The agent drafts emails, follow-ups, and meeting invites into a review surface; nothing is sent, and no irreversible CRM write to a customer-visible field happens, until the rep approves. Internal, low-risk writes (logging an internal note, updating a private next-step) can run with lighter review. Observability via LangSmith or Langfuse traces each run end to end, and semantic caching reuses account-research and product-answer results to cut latency and cost on repeated questions. ### Request flow - 1. The rep makes a request ("prep me for the Acme renewal call") and the router classifies intent and selects the tools and retrieval scope. - 2. The agent reads live account state from CRM via function calls — open opportunities, contacts, stage, recent activity — to ground its reasoning in the real deal. - 3. It retrieves supporting product and deal knowledge through RAG: specs, pricing rules, security docs, and battlecards relevant to the account. - 4. The agent drafts the requested artifact (brief, email, follow-up, next-best-action) with citations or attached sources for every product or pricing claim. - 5. Guardrails check the draft for unverifiable claims, sensitive data, and policy violations; outbound drafts route to the human-approval gate for rep review and edits. - 6. On approval the action executes — email sent, meeting scheduled, activity logged to CRM — and the full run is traced in observability for later audit and evaluation. ### Components - Orchestration loop with intent router - CRM integration via function calling (reads and writes) - RAG index over product and deal knowledge - Email and calendar tools - Guardrails and claim-verification layer - Human-approval gate for outbound actions - Observability and semantic caching ### Reference scenario - Context: A mid-market B2B software vendor equips its sales team with a copilot to reduce administrative load and improve the quality of account research and outreach. This is an illustrative, vendor-neutral blueprint, not a description of a specific deployment. - Scenario: Reps ask the copilot to prepare meeting briefs, draft renewal and prospecting emails, summarize account history, log calls, and recommend next-best-actions. The copilot grounds every artifact in CRM state and the product knowledge base, and routes all customer-facing drafts to the rep for approval before sending. - Technology: An orchestration loop calls the CRM through function calling, retrieves product and deal data via RAG, and uses email and calendar tools. Guardrails verify product and pricing claims, a human-approval gate fronts outbound actions, and LangSmith or Langfuse provide tracing with semantic caching over repeated research. - Load: Assume a few hundred reps issuing tens of requests each per day, with bursts around quarter-end. Research and drafting requests dominate; outbound sends are comparatively rare because each passes through human review. - Results: All figures here are reference targets to size and instrument the system, not guarantees: teams should expect to measure their own outcomes on their own data. Plausible targets include reduced time spent on pre-call research and CRM logging, faster first-draft turnaround on outreach, and improved data hygiene — each to be validated against a baseline before and after rollout. ### Benefits - Reps spend less time on administrative work — research, logging, and drafting — and more time in live conversations. - Outreach and briefs are grounded in the company's real CRM and product data, improving relevance and consistency. - The human-approval gate keeps reps accountable for every customer-facing message while still accelerating their workflow. - Observability and tracing make the system auditable, so the team can inspect what was read, drafted, and sent. ### Risks - Hallucinated product features or pricing can leak into customer communication if grounding and claim verification are weak. - CRM write access creates the risk of corrupting pipeline data if guardrails or approval gates are misconfigured. - Over-automation can erode rep judgment and relationships if the copilot is allowed to send without genuine review. - Sensitive customer and deal data flowing through retrieval and prompts raises privacy and access-control concerns. ### KPIs - Time saved per rep on research and logging: Measure pre-call research and CRM-logging time before and after rollout; good looks like a clear, sustained reduction without data-quality loss. - Approval-gate edit and rejection rate: Track how often reps edit or reject drafts; a healthy, non-trivial rate shows reps are genuinely reviewing rather than rubber-stamping. - Grounding and claim-verification accuracy: Sample drafts and check product and pricing claims against authoritative sources; good means near-zero unverifiable claims reaching customers. - Pipeline and conversion impact: Compare conversion or progression for copilot-assisted versus baseline activity; attribute cautiously and look for honest, durable lift. - Cost and latency per assisted task: Track tokens, retrieval calls, and response time per request; semantic caching should hold cost and latency stable as usage grows. ### Cost & scaling - Cost is driven by retrieval volume and model calls per request; semantic caching of repeated account research and product answers is the main lever to contain it. - Read-heavy traffic (research, briefs, summaries) scales horizontally and benefits from caching; outbound writes are rarer and bounded by human review. - Quarter-end and campaign bursts require headroom in retrieval and inference capacity, plus rate limiting so a few heavy users do not starve others. - As the product catalog and CRM grow, retrieval freshness and index maintenance dominate operational cost more than raw inference. ### Observed failure modes - The agent asserts a feature or price that is outdated or simply wrong because retrieval missed the authoritative source. - Reps rubber-stamp drafts without reading them, turning the approval gate into a formality that ships errors. - Stale or partial CRM reads cause the agent to brief on a deal stage or contact that no longer reflects reality. - The orchestration loop over-plans or loops on ambiguous requests, burning tokens and latency without converging. ### Lessons learned - Separate drafting from sending early; the human-approval gate is the single most important safety control for outbound work. - Treat product and pricing claims as citation-required: if the source cannot be attached, the claim should not ship. - Instrument from day one — without tracing you cannot tell whether the copilot helped or just generated more activity. - Scope CRM writes tightly and reversibly, keeping high-risk, customer-visible changes behind explicit rep confirmation. ### Technologies - CRM integration (function calling) - RAG over product & deal data - Email & calendar tools - Orchestration loop - Guardrails - Observability (LangSmith / Langfuse) ### Examples - A rep asks for a renewal-call brief; the copilot reads the account from CRM, retrieves the relevant contract and product docs, and drafts talking points with sources. - After a discovery call, the rep dictates notes; the copilot logs a structured activity to CRM and proposes a follow-up email that the rep edits and approves before sending. - The copilot scans a rep's book of business and surfaces next-best-actions — accounts gone quiet, expiring contracts, upsell signals — each linked to the CRM record behind it. ### FAQs **Can the copilot send emails to customers on its own?** No. By design it only drafts; every customer-facing email, follow-up, or invite passes through a human-approval gate where the rep reviews, edits, and approves before anything is sent. **How do you stop it from inventing product features or prices?** Factual claims about products or pricing must be grounded in retrieved authoritative sources and carry a citation. Guardrails reject outputs that assert unverifiable specifics, and drafts are sampled to verify accuracy. **How do you measure whether it actually helps?** Through honest metrics — time saved on research and logging, the approval-gate edit rate, grounding accuracy, and cautiously attributed pipeline impact — compared against a baseline rather than raw activity counts. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework Related: customer-service-agent, enterprise-knowledge-assistant --- ## ARCH-004 — AI Workforce URL: https://santismm.com/en/architectures/ai-workforce | API: https://santismm.com/api/architectures/ai-workforce Category: workforce | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: medium | Source: industry_observation, paper Patterns used: supervisor-agent, orchestrator-workers, goal-decomposition, human-escalation, task-prioritization | Builds on: multi-agent-architecture, ai-agent, agentic-evaluation, ai-observability, ai-governance **Definition.** The AI workforce architecture is a governed, multi-agent system in which a supervisor decomposes goals and delegates prioritized subtasks to specialized agents that share state, escalate to humans, and are observed end to end. An AI workforce is an orchestrated team of specialized agents that collaborate on multi-step business processes under a supervisor. The supervisor decomposes a goal, prioritizes and delegates subtasks to specialist agents (research, drafting, QA), and they share state through a common store. Agents escalate to humans when out of depth, and every step is observable and governed. Treat it as managing digital workers: success depends less on any single agent and more on coordination, clear ownership via an agent registry, human oversight, and evaluating the whole team rather than parts. ### Key concepts - A supervisor decomposes goals and delegates to specialist agents rather than one agent doing everything. - Shared state and an agent registry give the team memory, clear ownership, and discoverable capabilities. - Human oversight, escalation, and governance bound the blast radius when agents act incorrectly. - You evaluate and measure the workforce as a system, not each agent in isolation. ### Architecture At the center sits a supervisor (or orchestrator) that receives a business goal, decomposes it into a plan of subtasks, prioritizes them, and routes each to the specialist agent best suited to it. Specialists — for example research, drafting, QA, and tooling agents — are registered in an agent registry that records each agent's capabilities, inputs, outputs, cost, and owner, so the supervisor can discover and select them and humans can hold someone accountable for behavior. Agents do not pass everything through prompts. They read and write a shared memory or state store that holds the evolving job context, intermediate artifacts, and decisions. This shared state is what turns a loose collection of agents into a team: it preserves continuity across steps, lets agents build on each other's work, and gives observability and audit a single source of truth. Guardrails sit between agents and tools or data to constrain actions, validate outputs, and prevent unsafe operations. Wrapping the whole system are human oversight and governance. Defined escalation paths let an agent hand off to a person when confidence is low or a task is high-stakes, and approval gates require human sign-off before irreversible actions. Observability (traces, evaluations, cost and latency per agent and per job) makes the team's behavior legible. Because failures in one agent can cascade, the architecture deliberately limits the blast radius through scoping, timeouts, and circuit-breakers. ### Request flow - 1. Intake: a business goal or job enters the system and the supervisor records it with context, priority, and ownership. - 2. Decompose and prioritize: the supervisor breaks the goal into ordered subtasks and ranks them, considering dependencies and urgency. - 3. Delegate: each subtask is routed to a specialist agent selected from the agent registry by capability and cost. - 4. Execute with shared state: agents work against guardrailed tools, reading and writing the shared store so later agents build on earlier results. - 5. Escalate or approve: when confidence is low or stakes are high, an agent escalates to a human or waits at an approval gate. - 6. Assemble, verify, and close: a QA step checks the combined output, the supervisor finalizes the job, and traces and metrics are emitted for evaluation. ### Components - Supervisor / orchestrator that plans and delegates - Agent registry of specialists with owners and capabilities - Specialist agents (research, drafting, QA, tooling) - Shared memory / state store for job context - Guardrails over tools, data, and outputs - Human oversight: escalation paths and approval gates - Observability and evaluation across the workforce ### Reference scenario - Context: A mid-size enterprise wants to automate the drafting of customer-facing policy responses that today require a research step, a drafting step, and a compliance review before a human signs off. - Scenario: A goal enters the system; the supervisor decomposes it into research, draft, and QA subtasks, delegates each to a specialist agent, and routes anything compliance-sensitive to a human approval gate before release. - Technology: LangGraph for multi-agent orchestration, an agent registry for capability discovery and ownership, a shared state store for job context, guardrails over retrieval and tools, and LangSmith or Langfuse for tracing and evaluation. - Load: Illustrative volume of a few thousand jobs per week, with bursts during business hours and a long tail of complex jobs that require human escalation. - Results: All figures here are reference targets to instrument and measure in your own environment, not guarantees: aim to track end-to-end job success, escalation rate, rework rate, and cost per completed job, and validate them against a human baseline before claiming productivity gains. ### Benefits - Specialization lets each agent be simpler, better evaluated, and easier to own than one monolithic agent. - A supervisor with prioritization handles multi-step, dependency-laden work that a single agent struggles to sequence. - Shared state and a registry make the team auditable, with clear ownership and discoverable capabilities. - Human oversight and governance let you adopt automation incrementally while bounding risk. ### Risks - Coordination cost grows non-linearly; more agents and handoffs can add latency and error compounding. - An error in one agent can cascade across the team, widening the failure blast radius. - Without an agent registry and clear ownership, behavior becomes opaque and no one is accountable. - Productivity claims can be illusory if measured per-task rather than against an honest end-to-end human baseline. ### KPIs - End-to-end job success rate: Share of jobs completed correctly without human correction; the headline measure of whether the team actually works. - Escalation rate: Fraction of jobs handed to humans; too high means weak automation, too low may mean unsafe over-automation. - Rework / correction rate: How often outputs are sent back or fixed; rising rework signals cascading errors or weak QA. - Cost per completed job: Total model, tool, and human-review cost per finished job; the real unit economics behind productivity claims. - Coordination overhead: Added latency and token cost from handoffs versus a single-agent baseline; watch it grow with team size. ### Cost & scaling - Scale by adding specialist agents behind the registry, but treat each addition as new coordination surface to evaluate. - Partition work so independent subtasks run in parallel while preserving shared-state consistency. - Apply timeouts, retries with backoff, and circuit-breakers so a slow or failing agent cannot stall the whole job. - Tier human oversight: lighter review for low-stakes jobs, mandatory approval gates for irreversible or sensitive ones. ### Observed failure modes - Cascading failure: a wrong intermediate result is trusted downstream and corrupts the final output. - Coordination deadlock or loops: agents wait on each other or re-delegate the same subtask indefinitely. - Lost or stale shared state: agents act on out-of-date context and produce inconsistent results. - Escalation gaps: an agent proceeds on a high-stakes task instead of handing off to a human. ### Lessons learned - Start with the smallest team that works and add specialists only when a single agent demonstrably cannot cope. - Invest early in the agent registry, ownership, and shared-state contracts; they are what make the system governable. - Evaluate the whole workforce end to end, not just individual agents, because coordination is where it breaks. - Design escalation and blast-radius limits from day one rather than bolting on governance after an incident. ### Technologies - Multi-agent orchestration (LangGraph) - Agent registry - Shared memory / state store - Human oversight & approvals - Guardrails - Observability (LangSmith / Langfuse) ### Examples - A research-and-report workflow where a supervisor delegates gathering, synthesis, and fact-check QA to separate agents. - A customer-response pipeline that drafts replies but routes compliance-sensitive cases to a human approval gate. - An operations back-office team of agents that triage tickets, prepare actions, and escalate exceptions to staff. ### FAQs **How is this different from a single AI agent?** A single agent does all reasoning and tool use itself. An AI workforce is many coordinated agents under a supervisor that decomposes goals, delegates to specialists, shares state, and governs the whole team; the hard problems are coordination, ownership, and blast radius rather than any one agent's capability. **Do more agents always make the system better?** No. Each added agent introduces handoffs, latency, and new ways to fail. Add specialists only when a simpler team demonstrably cannot do the work, and measure coordination overhead so the cost of orchestration does not exceed its benefit. **How do we measure real productivity gains?** Measure end to end against an honest human baseline: job success, escalation, rework, and cost per completed job. Per-task speedups can hide downstream corrections and review time, so only count gains that survive full end-to-end evaluation. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework Related: customer-service-agent, enterprise-knowledge-assistant --- ## ARCH-005 — Operations Center URL: https://santismm.com/en/architectures/operations-center | API: https://santismm.com/api/architectures/operations-center Category: operations | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: medium | Source: industry_observation Patterns used: routing, recovery-strategy, human-escalation, evaluator-optimizer, human-approval-gate | Builds on: ai-agent, tool-use, ai-observability, guardrails, agentic-evaluation **Definition.** The operations center architecture is an agentic AIOps pattern that triages alerts, diagnoses root cause, and runs only approved runbook remediations while gating risky actions behind human approval. An operations center is an agentic AIOps system that watches monitoring signals and alerts, correlates and triages them, diagnoses likely root cause, and executes only vetted runbook remediations — keeping destructive or novel actions behind human approval. It cuts alert fatigue and shortens mean time to resolution by automating safe, read-mostly diagnostics while escalating risky writes to on-call engineers. Every action is audited and reversible. Success is measured honestly with MTTR, false-action rate, and escalation precision, not by automation volume. ### Key concepts - Read-mostly diagnostics run automatically; write or destructive actions require an explicit human approval gate. - Alert correlation collapses noisy, redundant signals into a single incident to reduce fatigue. - Remediation is bounded to vetted, versioned runbooks with safe rollback — never improvised actions. - Every decision and action is captured in an immutable audit trail for review and learning. ### Architecture Signals enter through an ingestion and normalization layer that unifies metrics, logs, traces, and alerts from heterogeneous monitoring tools into a common event schema. A correlation engine groups related signals by service, time window, and dependency graph so that a single underlying fault surfaces as one incident rather than dozens of duplicate pages. A triage-and-diagnosis agent reasons over the correlated incident, pulls additional context through read-only tools (dashboards, recent deploys, topology, prior incidents), and proposes a likely root cause with a confidence estimate. A router classifies each incident by severity, blast radius, and whether a matching vetted runbook exists, then chooses between automated remediation, human approval, or direct escalation. Remediation executes through a guarded action layer where read-mostly steps run automatically but any write, restart, scale, or rollback passes a human approval gate. An evaluator checks outcomes against expected health signals and can trigger safe rollback. Observability and an immutable audit trail wrap every step, feeding a feedback loop that improves runbooks and routing over time. ### Request flow - 1. A monitoring tool fires an alert; the ingestion layer normalizes it and the correlation engine merges it with related signals into one incident. - 2. The triage agent enriches the incident with read-only context — recent deploys, topology, dashboards, and similar past incidents. - 3. The diagnosis agent proposes a likely root cause with a confidence score and identifies whether a vetted runbook matches the symptom. - 4. The router decides the path: auto-run safe diagnostics, request human approval for write actions, or escalate novel or low-confidence cases to on-call. - 5. Approved remediation runs step by step from the runbook; the evaluator watches health signals and rolls back automatically if recovery fails. - 6. The incident is resolved or handed to a human, and the full timeline, decisions, and actions are written to the audit trail for review. ### Components - Signal ingestion and normalization layer - Alert correlation and deduplication engine - Triage and root-cause diagnosis agent - Severity and runbook router - Guarded remediation layer with human approval gates - Outcome evaluator with safe rollback - Immutable audit trail and observability ### Reference scenario - Context: An illustrative mid-size SaaS provider runs dozens of microservices across two regions and is overwhelmed by redundant alerts during incidents, slowing response. - Scenario: During a partial database failover, the operations center correlates a burst of latency, error-rate, and timeout alerts into a single incident, diagnoses a connection-pool exhaustion as the likely cause, runs read-only checks automatically, and requests human approval before recycling pool workers from a vetted runbook. - Technology: Monitoring and alerting integrations feed a correlation engine and a triage agent; an incident management system tracks state; runbook automation executes approved steps; human approval gates and guardrails bound write actions; observability tooling captures traces. - Load: Reference planning figures only: roughly 4,000 raw alerts per day collapsing to a few hundred incidents, with peak bursts of several hundred signals within minutes during major events. - Results: Reference targets to measure, not guarantees: aim to reduce duplicate pages through correlation, shorten MTTR for runbook-covered incidents, and keep the false-action rate near zero by gating all writes. Validate every figure against your own baseline before relying on it. ### Benefits - Correlation and deduplication sharply reduce alert fatigue and page volume for on-call staff. - Automating safe, read-mostly diagnostics shortens mean time to resolution for well-understood incidents. - Human approval gates keep destructive actions safe while still accelerating low-risk remediation. - A complete audit trail improves postmortems, compliance, and continuous runbook improvement. ### Risks - Over-trusting confidence scores can let a wrong diagnosis drive an inappropriate remediation. - Automating beyond vetted runbooks risks novel, untested actions causing wider outages. - Poorly tuned correlation can either merge unrelated incidents or fail to collapse duplicates. - Approval-gate fatigue may push engineers to rubber-stamp requests without real review. ### KPIs - Mean time to resolution (MTTR): Track separately for runbook-covered versus escalated incidents; good looks like a steady decline for covered cases without regressions elsewhere. - False-action rate: Share of automated remediations that were wrong or harmful; good is near zero, sustained by tight write-action gating. - Alert-to-incident compression: Ratio of raw alerts to correlated incidents; good means far fewer pages without hiding real distinct problems. - Escalation precision: Fraction of escalations that genuinely needed a human; good avoids both over-escalation fatigue and missed risky cases. - Rollback success rate: Share of failed remediations that rolled back cleanly to a safe state; good is consistently high with no lingering side effects. ### Cost & scaling - Partition correlation and routing by service domain or region so incident volume scales horizontally. - Keep runbooks versioned and independently testable so new automations can be added safely. - Rate-limit and back-pressure ingestion to survive alert storms without losing audit fidelity. - Expand automation coverage gradually, promoting runbooks from suggest-only to gated execution as confidence grows. ### Observed failure modes - Alert storms overwhelm correlation, producing either one giant incident or a flood of fragments. - A flawed runbook executes a harmful action that the evaluator fails to detect and roll back. - The agent escalates everything, recreating the alert fatigue it was meant to remove. - Stale topology or context data leads diagnosis toward the wrong root cause. ### Lessons learned - Default to read-mostly automation and require human approval for every write or destructive action. - Never auto-remediate beyond vetted, versioned runbooks with tested, safe rollback paths. - Measure MTTR and false-action rate honestly rather than celebrating automation volume. - Invest early in correlation quality; noisy incidents poison both diagnosis and human trust. ### Technologies - Monitoring & alerting integration - Runbook automation tools - Incident management systems - Human approval gates - Guardrails - Observability (LangSmith / Langfuse) ### Examples - Correlating a deploy-triggered error spike into one incident and recommending a gated rollback of the latest release. - Auto-running read-only disk, memory, and connection diagnostics, then requesting approval to recycle a saturated service. - Escalating a novel, low-confidence networking anomaly directly to on-call with enriched context instead of guessing. ### FAQs **Why not let the agent fix everything automatically?** Because destructive or novel actions can cause wider outages. The pattern automates safe, read-mostly diagnostics and gates every write behind human approval and a vetted runbook. **How does it reduce alert fatigue?** A correlation engine deduplicates and groups related signals into a single incident, so one underlying fault produces one page instead of dozens of redundant alerts. **What happens when a remediation goes wrong?** An evaluator compares outcomes to expected health signals and triggers a safe, tested rollback, while the full timeline is captured in the audit trail for postmortem review. ### References - Anthropic — Building Effective Agents (2024) — https://www.anthropic.com/research/building-effective-agents - Google SRE Book — Managing Incidents — https://sre.google/sre-book/managing-incidents/ Related: customer-service-agent, ai-workforce ================================================================= # ENTERPRISE AI GOVERNANCE ================================================================= ## GOV-001 — EU AI Act URL: https://santismm.com/en/governance/eu-ai-act | API: https://santismm.com/api/governance/eu-ai-act Category: regulation | Frameworks: EU AI Act | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: paper, industry_observation Operationalized by: human-approval-gate | Related knowledge: ai-governance, guardrails, human-in-the-loop, ai-observability **Definition.** The EU AI Act is a horizontal European regulation that governs AI by risk tier, imposing obligations on providers and deployers proportional to the risk an AI system poses to health, safety and fundamental rights. The EU AI Act is the European Union's comprehensive, risk-based law for artificial intelligence. It sorts AI systems into risk tiers — unacceptable (banned), high-risk (strict obligations), limited-risk (transparency duties) and minimal-risk — and adds specific obligations for general-purpose AI models. It applies extraterritorially to anyone placing AI on the EU market and phases in over several years, with penalties reaching up to 7% of global annual turnover for the most serious breaches. ### Scope Providers and deployers that place AI systems or general-purpose AI models on the EU market or whose output is used in the EU — regardless of where they are established. Some uses (e.g. purely personal, certain research) are out of scope. ### Key requirements - Risk-based tiers: unacceptable (prohibited), high-risk, limited-risk (transparency), minimal-risk. - Prohibited practices include social scoring and certain biometric and manipulative uses. - High-risk systems require risk management, data governance, technical documentation, logging, human oversight, accuracy/robustness and post-market monitoring. - General-purpose AI (GPAI) models carry their own transparency and, for systemic-risk models, additional obligations. - Transparency duties: users must be told when they interact with AI, and synthetic content must be marked. - Phased application with significant penalties for non-compliance. ### Controls - Risk classification: Determine each system's tier first — it decides every other obligation. Misclassifying a high-risk system is the costliest early mistake. - Human oversight (Art. 14): High-risk systems must be overseeable by a person who can intervene or stop them — the regulatory basis for human-approval gates. - Technical documentation & logging: Maintain documentation and automatic event logs so the system is traceable and auditable across its lifecycle. - Transparency to users: Disclose AI interaction and label AI-generated or manipulated content (deepfakes). - Post-market monitoring: Monitor performance in the field and report serious incidents; governance does not end at deployment. ### Checklist - Inventory your AI systems and classify each by risk tier. - Confirm none fall under prohibited practices. - For high-risk systems, stand up risk management, data governance and technical documentation. - Implement human oversight with the ability to intervene or stop the system. - Enable logging/traceability and define post-market monitoring and incident reporting. - Add user-facing transparency and content labelling where required. - Track the phased application dates that apply to your systems. ### Common pitfalls - Assuming the Act doesn't apply because you're outside the EU — it is extraterritorial. - Treating GPAI obligations as identical to AI-system obligations; they are distinct. - Bolting on human oversight that can't actually intervene in time. - Under-documenting: missing technical documentation and logs is a common gap. ### Examples - A hiring screening tool classified as high-risk, requiring documentation, human oversight and monitoring. - A chatbot adding a clear 'you are talking to an AI' disclosure to meet transparency duties. - A GPAI provider publishing model documentation and a training-data summary. ### FAQs **Does the EU AI Act apply to companies outside the EU?** Yes. It applies to providers and deployers whose AI systems are placed on the EU market or whose output is used in the EU, regardless of where the company is established. **What is a 'high-risk' AI system?** Systems used in sensitive areas (e.g. employment, credit, critical infrastructure, certain biometrics) or as safety components of regulated products. They carry the strictest obligations short of prohibition. **How does it relate to human-in-the-loop?** Article 14 requires effective human oversight for high-risk systems — a person able to understand, intervene in or stop the system. The human-approval-gate pattern is one way to implement it. ### References - European Union — Artificial Intelligence Act (Regulation (EU) 2024/1689) — https://artificialintelligenceact.eu/ - EU AI Act — Article 14 (Human oversight) — https://artificialintelligenceact.eu/article/14/ - European Commission — AI Act overview — https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai Related: iso-42001, nist-ai-rmf, agentic-ai-governance-checklist --- ## GOV-002 — ISO/IEC 42001 URL: https://santismm.com/en/governance/iso-42001 | API: https://santismm.com/api/governance/iso-42001 Category: standard | Frameworks: ISO/IEC 42001 | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: paper, industry_observation Operationalized by: human-approval-gate | Related knowledge: ai-governance, agentic-evaluation, ai-observability, guardrails **Definition.** ISO/IEC 42001 is a management-system standard that specifies requirements for establishing and continually improving an Artificial Intelligence Management System (AIMS) across an organization's AI lifecycle. ISO/IEC 42001:2023 is the first international, certifiable standard for an AI management system (AIMS). Like ISO 27001 for information security, it defines how an organization should establish, implement, maintain and continually improve the way it governs AI — through a policy, defined roles, risk and impact assessments, a set of controls, and a Plan-Do-Check-Act improvement cycle. It is voluntary and certifiable, giving organizations a recognized way to demonstrate responsible AI management. ### Scope Any organization that provides or uses AI, of any size or sector. It governs the management system around AI — not a specific product — so it complements product- or risk-specific frameworks rather than replacing them. ### Key requirements - A certifiable AI management system, structured like other ISO management standards. - Requires an AI policy, leadership commitment and clearly assigned roles and responsibilities. - Centres on AI risk assessment and AI system impact assessment. - Provides a reference set of controls (Annex A) and implementation guidance (Annex B). - Built on the Plan-Do-Check-Act cycle for continual improvement. - Complements regulation (EU AI Act) and risk frameworks (NIST AI RMF). ### Controls - AI policy & governance roles: Establish an organizational AI policy and assign accountable owners — governance starts with leadership, not tooling. - AI risk assessment: Systematically identify, analyse and treat risks across the AI lifecycle, and keep the assessment current. - AI system impact assessment: Assess impacts on individuals and society (fairness, safety, rights), not just technical risk. - Lifecycle controls (Annex A): Apply controls for data, design, deployment and operation, selecting those relevant to your context. - Continual improvement (PDCA): Audit, review and improve the management system on a cycle, so governance keeps pace with change. ### Checklist - Define the AIMS scope and an organizational AI policy. - Assign governance roles, responsibilities and leadership accountability. - Run AI risk assessments and AI system impact assessments. - Select and implement the relevant Annex A controls. - Document objectives, processes and evidence of operation. - Establish internal audit and management review. - Run the Plan-Do-Check-Act cycle and pursue certification if desired. ### Common pitfalls - Treating it as a one-off project rather than a continuing management system. - Documenting a policy nobody operates against day to day. - Confusing it with EU AI Act compliance — certification is not legal conformity. - Skipping impact assessment and reducing it to technical risk only. ### Examples - A company standing up an AIMS to govern all its AI use under one policy and risk process. - An impact assessment surfacing a fairness risk before a model ships. - An annual internal audit and management review closing governance gaps. ### FAQs **Is ISO/IEC 42001 the same as complying with the EU AI Act?** No. The standard is a voluntary, certifiable management system; the EU AI Act is binding law. A well-run AIMS supports legal compliance but does not by itself satisfy it. **Can you get certified?** Yes. Like ISO 27001, an accredited body can audit and certify an organization's AI management system against the standard. **How does it relate to NIST AI RMF?** They are complementary: NIST AI RMF gives a risk-management framework and trustworthiness characteristics; ISO/IEC 42001 gives the certifiable management-system structure to operate governance continuously. ### References - ISO/IEC 42001:2023 — AI management system — https://www.iso.org/standard/81230.html - ISO — What is an AI management system? — https://www.iso.org/artificial-intelligence/ai-management-systems Related: eu-ai-act, nist-ai-rmf, agentic-ai-governance-checklist --- ## GOV-003 — NIST AI Risk Management Framework URL: https://santismm.com/en/governance/nist-ai-rmf | API: https://santismm.com/api/governance/nist-ai-rmf Category: framework | Frameworks: NIST AI RMF | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: paper, industry_observation Operationalized by: human-approval-gate, evaluator-optimizer | Related knowledge: ai-governance, agentic-evaluation, ai-observability, guardrails **Definition.** The NIST AI Risk Management Framework is a voluntary framework that helps organizations govern, map, measure and manage the risks of AI systems while pursuing the characteristics of trustworthy AI. The NIST AI RMF 1.0 is a voluntary, widely-adopted framework for managing AI risk across the lifecycle. It is organized around four functions — Govern, Map, Measure and Manage — and a set of characteristics of trustworthy AI (valid and reliable, safe, secure and resilient, accountable and transparent, explainable, privacy-enhanced, and fair with harmful bias managed). A companion Generative AI Profile adapts it to GenAI risks. Unlike the EU AI Act it is not law, but it is a common backbone for operational AI governance. ### Scope Any organization designing, developing, deploying or using AI, in any sector. It is voluntary and outcome-focused, designed to be tailored to context and used alongside standards and regulation. ### Key requirements - Four core functions: Govern (culture & accountability), Map (context & risks), Measure (assess & track), Manage (prioritize & respond). - Govern is cross-cutting — it underpins the other three. - Defines characteristics of trustworthy AI to aim for, not just risks to avoid. - A companion Generative AI Profile (NIST AI 600-1) addresses GenAI-specific risks. - Voluntary and flexible — meant to be tailored, not certified against. - Pairs well with ISO/IEC 42001 (management system) and the EU AI Act (law). ### Controls - Govern: Establish the policies, accountability, culture and roles that make risk management real — the foundation the other functions stand on. - Map: Establish context: intended use, stakeholders, and the risks and impacts of the AI system before building. - Measure: Use quantitative and qualitative methods to assess, benchmark and monitor risk and trustworthiness — you can't manage what you don't measure. - Manage: Prioritize, respond to and track risks over time, including incident response and decommissioning. - Trustworthiness characteristics: Steer toward valid, safe, secure, accountable, explainable, privacy-enhanced and fair outcomes as explicit design targets. ### Checklist - Stand up the Govern function: policy, accountability and roles. - Map each system's context, intended use, stakeholders and risks. - Define metrics and Measure validity, safety, security, bias and robustness. - Manage: prioritize risks, plan responses and track them over time. - Apply the Generative AI Profile for GenAI systems. - Set incident response and monitoring for deployed systems. - Map the framework to your obligations under ISO 42001 and the EU AI Act. ### Common pitfalls - Doing Map and Measure but neglecting Govern, so nothing is accountable. - Measuring what's easy instead of what matters for trustworthiness. - Treating it as a checklist rather than a continuous risk practice. - Ignoring the Generative AI Profile for LLM and agentic systems. ### Examples - A team using Map to document an agent's intended use and stakeholders before building. - A Measure step benchmarking a model for bias and robustness against an eval set. - A Manage process with incident response for a deployed GenAI assistant. ### FAQs **Is the NIST AI RMF mandatory?** No. It is a voluntary framework. But it is widely adopted as a common language and backbone for operational AI risk management, and often referenced in policy and procurement. **What are the four functions?** Govern, Map, Measure and Manage. Govern is cross-cutting and supports the other three, which run across the AI lifecycle. **How does it handle generative AI?** Through the companion Generative AI Profile (NIST AI 600-1), which identifies GenAI-specific risks and suggested actions mapped to the four functions. ### References - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework - NIST AI 600-1 — Generative AI Profile — https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence Related: eu-ai-act, iso-42001, agentic-ai-governance-checklist --- ## GOV-004 — Agentic AI Governance Checklist URL: https://santismm.com/en/governance/agentic-ai-governance-checklist | API: https://santismm.com/api/governance/agentic-ai-governance-checklist Category: playbook | Frameworks: EU AI Act, ISO/IEC 42001, NIST AI RMF | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: medium | Source: industry_observation, personal_experience Operationalized by: human-approval-gate, evaluator-optimizer, reflection, routing | Related knowledge: ai-governance, guardrails, human-in-the-loop, ai-observability, agentic-evaluation, prompt-injection **Definition.** The agentic AI governance checklist is an operational control set that turns AI governance frameworks into concrete, implementable requirements for autonomous agents acting in production. A practical, vendor-neutral checklist for governing agentic AI in the enterprise — translating the principles of the EU AI Act, ISO/IEC 42001 and NIST AI RMF into concrete controls you can implement in a harness. It covers human oversight, guardrails, audit logging, evaluation, access control, prompt-injection defence and incident response, and maps each control to the patterns and knowledge units that operationalize it. Use it as a readiness gate before letting an agent act in production. ### Scope Teams building or deploying autonomous or semi-autonomous agents that use tools, act on systems, or make consequential decisions. It is a practical companion to the formal frameworks, not a substitute for legal advice. ### Key requirements - Human oversight by risk: gate high-impact, irreversible or regulated actions for human approval. - Guardrails on inputs and outputs, including prompt-injection and PII defence. - Full audit logging and observability so every action is traceable. - Evaluation before and after deployment, against a maintained eval set. - Least-privilege access for tools and data the agent can reach. - A defined incident response and kill-switch for agents in production. ### Controls - Human approval gates: Route high-impact actions through a human checkpoint (EU AI Act Art. 14). Implements the human-approval-gate pattern. - Guardrails: Validate and constrain inputs and outputs; defend against prompt injection and block out-of-policy actions. - Audit logging & observability: Trace every decision, tool call and action so the agent is reviewable and incidents are reconstructable. - Evaluation harness: Score behaviour against an eval set before shipping and monitor for regressions after — NIST 'Measure'. - Least-privilege access: Scope the tools, data and permissions an agent can reach to the minimum its task requires. - Incident response & kill-switch: Define how to detect, stop and remediate a misbehaving agent, including a way to halt it immediately. ### Checklist - Classify the agent's risk and identify which actions need human approval. - Implement guardrails for inputs/outputs and prompt-injection defence. - Enable end-to-end audit logging and observability. - Stand up an evaluation set and run it pre-deployment and continuously. - Apply least-privilege scoping to tools, data and credentials. - Define incident response, monitoring thresholds and a kill-switch. - Map each control to your obligations under the EU AI Act, ISO 42001 and NIST AI RMF. - Document ownership and review the agent on a schedule. ### Common pitfalls - Granting an agent broad tool/data access 'to be safe', creating a large blast radius. - Gating everything (approval fatigue) or nothing (no oversight) instead of gating by risk. - Shipping without an eval set, so quality and safety are unmeasured. - No kill-switch or incident plan when an agent misbehaves in production. - Ignoring prompt injection as an attack surface for tool-using agents. ### Production evidence - Context: Teams putting an autonomous or semi-autonomous agent into production where it uses tools and takes consequential actions. - Scenario: Before go-live, the team runs the checklist as a readiness gate: classify the agent's risk, gate high-impact actions for human approval, add guardrails and prompt-injection defence, enable audit logging and observability, stand up an evaluation set, scope least-privilege access, and define incident response and a kill-switch. - Technology: A harness combining a human-approval gate, guardrails, audit logging/observability, an evaluation harness and scoped tool/credential access. - Load: Applied per agent before deployment and re-reviewed on a schedule; the heaviest control (human approval) is reserved for the small set of high-impact actions. - Results: Observed pattern: teams that gate by risk, enforce least privilege and instrument from day one contain the blast radius of agent errors; those that grant broad access 'to be safe' or ship without evals discover failures in production. Measure escalation appropriateness, false-action rate and mean time to detect. ### Lessons learned - Treat the checklist as a readiness gate, not a one-time audit — re-run it as the agent's tools and autonomy grow. - Least-privilege access and risk-based human approval bound the blast radius more than any single guardrail. - Without an evaluation set and audit logging in place before launch, you cannot tell a safe agent from a lucky one. - Map each control to a concrete owner; governance without accountability is just documentation. ### Examples - An agent whose refund action is gated for human approval while read-only lookups run freely. - A guardrail blocking a prompt-injected instruction to exfiltrate data via a tool. - An evaluation run catching a safety regression before an agent update ships. ### FAQs **Is this a substitute for the EU AI Act or ISO 42001?** No. It is a practical control set that operationalizes their principles for agents. Use it alongside the formal frameworks and legal advice, not instead of them. **Which control matters most for autonomous agents?** Risk-based human oversight plus least-privilege access and audit logging — together they bound what an agent can do and make every action accountable. **How does it connect to the patterns library?** Each control maps to patterns that implement it — human-approval-gate for oversight, reflection and evaluator-optimizer for quality — and to knowledge units like guardrails and AI observability. ### References - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework - EU AI Act — Article 14 (Human oversight) — https://artificialintelligenceact.eu/article/14/ - OWASP — Top 10 for LLM Applications — https://owasp.org/www-project-top-10-for-large-language-model-applications/ Related: eu-ai-act, iso-42001, nist-ai-rmf --- ## GOV-005 — Enterprise AI Governance Framework URL: https://santismm.com/en/governance/enterprise-ai-governance-framework | API: https://santismm.com/api/governance/enterprise-ai-governance-framework Category: framework | Frameworks: EU AI Act, ISO/IEC 42001, NIST AI RMF | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper Operationalized by: human-approval-gate | Related knowledge: ai-governance **Definition.** An enterprise AI governance framework is the system of principles, roles, processes and controls by which an organization directs and controls how it builds, procures and operates AI so that AI use stays lawful, safe, effective and aligned with its risk appetite. An umbrella operating model for governing AI across an organization. It defines the principles, accountability (RACI), AI risk taxonomy, lifecycle gates and policy hierarchy that keep AI use lawful, safe and aligned with risk appetite. It harmonizes the EU AI Act, ISO/IEC 42001 and NIST AI RMF into one internal program — comply once, reuse everywhere — and composes the agentic governance checklist as its concrete control set. Use it to give every production AI system a named owner, a risk tier and a gate that can actually block a non-compliant deployment. ### Scope Boards, AI governance committees, risk and compliance officers, and the product and engineering leaders who build and operate AI across the enterprise. It is an internal operating model that maps to named regulations and standards, not legal advice or a substitute for qualified counsel. ### Key requirements - Principles first: lawfulness, accountability, human oversight, transparency, fairness, safety and privacy by design anchor every policy. - RACI accountability: every governance activity has a clear Responsible, Accountable, Consulted and Informed map, and every production system has one named owner. - A risk taxonomy tiers use cases (unacceptable, high, limited, minimal) so control intensity is proportional to risk. - Lifecycle gates attach entry and exit checks to each stage, from propose through retire. - A policy hierarchy traces principles to policies, standards and concrete controls with named owners. - Comply once, reuse everywhere: external obligations map to internal controls a single time and are shared across regimes. ### Controls - AI governance committee with a charter: A standing body with published decision rights sets risk appetite and policy, so accountability is structural rather than ad hoc. - Named accountable system owner: Every production AI system has one human owner; this is the control most associated with incidents being caught and answered. - Risk-tiering procedure and central register: A documented taxonomy classifies each use case before build and records it in an inventory, surfacing shadow AI and calibrating controls. - Lifecycle entry and exit gates: Each stage from propose to retire has gates scaled to risk tier; an enforceable gate can block a non-compliant deployment. - Policy hierarchy mapped to controls: Principles trace to policies, standards and concrete controls so obligations are operational, not aspirational. - Independent audit and assurance: Periodic independent review tests that gates and controls actually hold, closing the loop back to the committee. ### Checklist - Stand up an AI governance committee with a published charter and decision rights. - Define the seven governing principles and trace every policy back to them. - Publish a risk taxonomy with tiers and enumerate prohibited use cases blocked at intake. - Build a central register of all AI systems with their tier and named owner. - Attach entry and exit gates to each lifecycle stage, scaled to risk tier. - Map the EU AI Act, ISO/IEC 42001 and NIST AI RMF to internal controls once and reuse them. - Adopt the agentic governance checklist as the concrete control set for high-risk systems. - Schedule re-tiering and independent audit on a recurring cadence. ### Common pitfalls - Governance as paperwork: policies exist but no gate can actually block a non-compliant deployment. - No accountable owner: systems ship with diffuse ownership and no one is answerable when they fail. - Uniform controls: every system gets the same heavyweight process, so teams route around governance. - Shadow AI: systems are built outside the register and stay invisible to risk. - Static tiering: a use case's risk is set once and never re-evaluated as autonomy or scope grows. ### Examples - A regulated enterprise routes every new agent through a tier-based intake gate before any build begins. - A high-risk customer-facing system gets the full control set and independent audit while a minimal-risk internal tool follows baseline hygiene only. - EU AI Act, ISO/IEC 42001 and NIST AI RMF obligations are mapped to one internal control library and reused across the portfolio. ### FAQs **Is this framework legal advice?** No. It is a professional operating model that maps to named regulations and standards. Consult qualified counsel for binding compliance decisions. **How does it relate to the agentic governance checklist and the specific regimes?** This framework owns the structure — principles, roles, taxonomy and gates — and composes the EU AI Act, ISO/IEC 42001 and NIST AI RMF and the agentic checklist as its concrete controls. **What is the single highest-leverage element?** An enforceable gate plus a named accountable owner per system. Un-enforced policy is routed around within a quarter, and diffuse ownership means no one answers when a system fails. ### References - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework - ISO/IEC 42001:2023 — Artificial intelligence management system — https://www.iso.org/standard/81230.html - OECD AI Principles — https://oecd.ai/en/ai-principles Related: agentic-ai-governance-checklist, eu-ai-act, iso-42001, nist-ai-rmf --- ## GOV-006 — Audit Framework for Agentic Systems URL: https://santismm.com/en/governance/audit-framework-for-agentic-systems | API: https://santismm.com/api/governance/audit-framework-for-agentic-systems Category: framework | Frameworks: ISO/IEC 42001, NIST AI RMF | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: medium | Source: industry_observation, paper Operationalized by: human-approval-gate, evaluator-optimizer | Related knowledge: ai-observability, agentic-evaluation, ai-governance **Definition.** An agent audit is an independent, evidence-based examination that determines whether an agentic system operated within its authorized scope, controls and policies over a defined period, and whether the governance claims made about it are supported by reliable evidence. A practical, vendor-neutral framework for making an agent auditable and for auditing it. It defines the evidence an independent reviewer needs — immutable, correlated traces of every decision and tool call, model and version provenance, evaluation reports, approval and incident records — and how to test controls and sample high-volume runs. Each evidence type maps to ISO/IEC 42001 and NIST AI RMF so an auditor can verify the agent stayed within its governed bounds. Use it to design auditability in from the start, not as an afterthought. ### Scope Auditors, assurance and risk teams, and the system owners who must make their agents auditable by design. It applies to autonomous or semi-autonomous agents that use tools and act over time, and supports both internal assurance and external or regulatory audit. It is a practical companion to the formal frameworks, not a substitute for legal advice. ### Key requirements - Auditability by design: the agent must emit enough structured evidence at runtime to reconstruct any run later. - Every run produces an immutable, correlated trace: goal, each step and tool call with parameters and results, approvals, overrides and outcome. - Evidence must be complete, attributable, time-stamped and tamper-evident to hold evidentiary value. - Model and version provenance (prompt, tools, model) is captured per run so behaviour is tied to a known configuration. - Sampling is layered: risk-stratified, statistical, and 100% review of all exceptions, overrides, denials and incidents. - Audit evidence maps to ISO/IEC 42001 internal audit (Clause 9) and NIST AI RMF Measure and Manage functions. ### Controls - Immutable audit logs and traceability: Capture a correlated trace per run — agent identity and version, goal, each step, tool call, result, approvals and outcome — and protect it against alteration. Implements AI observability. - Model and version provenance: Record the exact prompt, tool set and model version behind each run so behaviour is attributable to a known, reproducible-by-config baseline. - Evaluation evidence: Retain pre-deployment and ongoing evaluation reports and gate results so the auditor can verify safety and quality were measured — NIST 'Measure'. - Control testing and sampling: Test each control against evidence using a documented sampling methodology: risk-stratified, statistical, and full exception review. - Approval and incident records: Capture human approvals, overrides and incidents with actor identity and timestamps. Implements the human-approval-gate pattern. - Attestations and findings management: System and control owners sign attestations backed by evidence; findings are tracked to closure with severity and deadlines. ### Checklist - Confirm every agent run produces a complete, correlated trace tied by a stable trace ID. - Verify logs are tamper-evident, time-stamped and retained for the full audit and regulatory window. - Check that each run records model and version provenance (prompt, tools, model version). - Retrieve a control mapping / Statement of Applicability linking each control to its evidence. - Retain and review pre-deployment and ongoing evaluation reports and gate outcomes. - Confirm approval, override and incident records exist with actor identity and timestamps. - Apply a documented sampling methodology — risk-stratified, statistical, and 100% exception coverage. - Track findings to closure with severity and deadlines, and collect signed owner attestations. ### Common pitfalls - Non-auditable agents: logging added as an afterthought, leaving gaps no audit can fill. - Mutable logs: records that could be altered, destroying their evidentiary value. - Sampling blind spots: random-only sampling that misses rare but catastrophic actions. - Attestation without evidence: owners signing off on controls they cannot demonstrate. - Findings graveyard: issues raised but never remediated or re-tested. ### Examples - An auditor reconstructs a disputed refund by following its correlated trace from goal to tool call to human approval. - A continuous-audit check alerts on a tool call outside the allow-list against the live log stream. - Retained evaluation reports map to NIST AI RMF Measure, evidencing that safety gates passed before deployment. ### FAQs **Can you audit a non-deterministic agent at all?** Yes — you audit the controls and the recorded behaviour, not the determinism. Complete, immutable traces make any specific run reconstructable even if it cannot be reproduced exactly. **How big should the audit sample be?** Large enough for your target assurance level statistically, plus 100% of exceptions and high-impact actions. Sampling never replaces full exception review. **How does audit evidence map to the frameworks?** Traces and logs support NIST AI RMF Measure and Manage and ISO/IEC 42001 Clause 9 internal audit; a control mapping or Statement of Applicability links each control to the evidence that proves it operated. ### References - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework - ISO/IEC 42001:2023 — AI management systems — https://www.iso.org/standard/81230.html - ISACA — Auditing Artificial Intelligence — https://www.isaca.org/resources/white-papers/2024/auditing-artificial-intelligence Related: agentic-ai-governance-checklist, nist-ai-rmf, iso-42001 --- ## GOV-007 — Human Oversight and Accountability Policy URL: https://santismm.com/en/governance/human-oversight-and-accountability-policy | API: https://santismm.com/api/governance/human-oversight-and-accountability-policy Category: playbook | Frameworks: EU AI Act | Updated: 2026-06-21 | Version: 1.0 Evidence: industry_observation | Confidence: high | Source: industry_observation, paper Operationalized by: human-approval-gate, human-escalation | Related knowledge: human-in-the-loop, ai-governance **Definition.** A human oversight and accountability policy is a binding rule set that assigns a named human to be answerable for each agent and guarantees a competent person can understand, intervene in, and stop its actions. An operational policy that turns EU AI Act Article 14 human oversight into practice for agentic AI. It assigns a named accountable owner per agent, sets the oversight level (in-the-loop, on-the-loop, out-of-the-loop) by risk, and defines intervention, override and stop authority plus escalation paths. It requires overseers to be competent and have time to act, and it guards against rubber-stamping and automation bias. It exists to prevent two failures: the absent human and the token human who cannot actually understand, override, or answer for what the agent does. ### Scope Every production or pilot agent that uses tools, acts on systems, or makes consequential decisions, and the system owners, approvers and operators who oversee them. It operationalizes Article 14; it is not a substitute for legal advice. ### Key requirements - Each agent has one named, accountable owner — accountability is never transferred to the model. - Oversight level is matched to risk: in-the-loop for high-impact or irreversible actions, on-the-loop for reversible high-volume actions, out-of-the-loop only for low-risk reversible tasks. - Every agent exposes tested reject, modify and stop (kill-switch) controls with the context needed for an informed decision. - Escalation thresholds route consequential decisions to humans by impact, irreversibility, rights or safety, confidence and novelty. - Overseers must be competent, intelligibly informed, and have genuine authority and time to act. - Automation bias and rubber-stamping are actively countered, not assumed away. ### Controls - Named accountable owner: Assign one human answerable for each agent's outcomes. 'The model decided' is not an acceptable account. - Risk-matched oversight level: Define in-the-loop, on-the-loop or out-of-the-loop per agent based on action impact and reversibility. Implements the human-approval-gate pattern for high-impact actions. - Override and stop authority: Expose tested reject, modify and stop controls; surface enough context for an informed override. The stop must be fast and reachable. - Escalation thresholds: Route decisions to humans when impact, irreversibility, rights/safety, low confidence or novelty thresholds are crossed. Implements the human-escalation pattern. - Overseer competence: Train and certify overseers on the agent's domain and limits so oversight is meaningful, not nominal. - Anti-rubber-stamping safeguards: Throttle and require justification for approvals; monitor approval time and override rates to detect automation bias. ### Checklist - Name one accountable owner for each production agent and record it. - Classify each agent's actions by impact and reversibility and assign an oversight level. - Implement and test reject, modify and stop (kill-switch) controls for every agent. - Ensure the agent surfaces intelligible context for any decision that needs oversight. - Define and configure escalation thresholds for impact, rights/safety, confidence and novelty. - Train overseers on the agent's domain and limits and keep their certification current. - Add anti-rubber-stamping safeguards and monitor approval time and override rates. - Log every approval and override with actor, reason and timestamp, and review thresholds on a schedule. ### Common pitfalls - Token oversight: a human clicks approve without the context, authority or time to actually evaluate the action. - Automation bias: approvers trust the agent so much they stop scrutinizing its output. - Diffuse accountability: no single named owner, so a failure has no answerable human. - Unreachable override: a stop control that is slow, hidden or never tested. - Threshold drift: escalation limits set once and never updated as the agent's scope grows. ### Examples - A finance agent whose payments above a spend cap require in-the-loop human approval, while reconciliations run on-the-loop. - A support agent that escalates to a human when its confidence is low or a request affects a customer's rights. - An incident where the named owner is held accountable and the override log shows who approved the action and why. ### FAQs **Does human oversight mean a human approves everything?** No. Oversight is tiered: in-the-loop for high-impact or irreversible actions, on-the-loop monitoring for reversible high-volume actions, and a human-in-command posture overall. The model scales to risk so oversight stays meaningful instead of becoming approval fatigue. **Can accountability sit with the AI vendor?** No. Vendor relationships are governed separately, but your named system owner remains accountable for how the agent is deployed and used. Automation is a tool, not a defense. **How do we prevent rubber-stamping and automation bias?** Surface intelligible context for each decision, throttle and require justification for approvals, monitor approval time and override rates, and keep overseers competent through training and rotation. ### References - EU AI Act — Article 14 (Human oversight) — https://artificialintelligenceact.eu/article/14/ - NIST — AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework - OECD — AI Principles — https://oecd.ai/en/ai-principles Related: eu-ai-act, agentic-ai-governance-checklist