We already use a tracing backend. Isn't this solved?

A backend gives you capture and storage. This pattern is about the three things a backend does not decide for you: whether one identifier threads the whole run, whether the fidelity is enough to reconstruct it without the source systems, and whether what you wrote down is safe to keep. Teams with excellent tooling routinely fail the reconstruction test.

Doesn't full capture conflict with data minimisation?

It would, if capture meant keeping everything raw. The control states both halves on purpose — enough to reconstruct, nothing that turns the log into the breach — and the way to hold both is field-level redaction at capture plus retention tied to the obligation that justifies the record. What you cannot do is decide the tension away by keeping everything and calling it compliance.

How much fidelity is enough?

Exactly enough to pass the reconstruction test on a run picked at random, and no more. That threshold is discoverable by trying it, which is why the test belongs in the routine rather than in an audit. Anything captured beyond it is cost and liability without a question it answers.

ReliabilityUpdated 2026-08-25 · Version 1.0

Correlated Run Trace

One identifier threads a whole agent run — inputs, model and version, every tool call with its arguments and outcome, the final action — into a record you can replay months later. The hard part is not capture. It is being complete enough to reconstruct the run and restrained enough that the trace store is not a second copy of the data it describes.

Evidence: Industry observationConfidence: MediumSource: Industry observationSource: Paper

Machine-readable: JSON

Definition

A correlated run trace is an immutable, single-identifier record of everything an agent run decided and did, captured at a fidelity that lets someone reconstruct the run end to end afterwards without access to the systems that produced it, and redacted so that the trace does not become an easier target than the source. It is a record of the agent's decisions, not a request log.

Problem

Agents are non-deterministic and multi-step, so 'what happened' cannot be inferred from the output. Most teams do log, and still cannot answer why a specific run did what it did: the steps are in different systems with no shared identifier, the prompt was recorded as a template reference that has since changed, or the model version was never captured, so nobody can tell whether the behaviour drifted or the model did.

When to use it

Any agent whose actions have consequences someone will later ask about: a regulated process where the trace is the audit trail and traceability is an obligation rather than a preference, an incident that needs a root cause, a customer disputing an outcome, or an evaluation loop that needs real failures to learn from.

Solution

Mint one run identifier at the entry point and propagate it through every step, service and retry. Correlation is the whole value: unlinked records of the same run are three logs, not a trace.

Record the rendered prompt, not a reference to it. A template id resolves to whatever the template says today, which is not what the model saw.

Capture the model and its version alongside every call. Without it, a behaviour change and a model change are indistinguishable after the fact.

Log tool calls as arguments plus outcome, including the failures and the retries. A trace that shows only successful calls describes a run that did not happen.

Redact at capture, not at read. Field-level rules that drop or hash secrets and personal data before the record is written, because a redaction applied at query time still leaves the raw value in storage.

Sample on the tail, not the head. Decide what to keep after the run finishes, so errors, escalations and anomalies are kept and the routine successes are the ones thinned.

Make the store append-only and give it the access controls of the most sensitive system it describes, not the ones a logging backend ships with.

Components

A run identifier minted at entry and propagated through every step, service, retry and asynchronous continuation.A span per step with a stable schema: inputs, model and version, tool name, arguments, outcome, latency, token usage.Field-level redaction applied at capture, with an explicit list of what is dropped, what is hashed and what is kept whole.Tail-based sampling that decides after the fact, so anomalous runs survive and routine ones are thinned.Append-only storage with retention aligned to the obligation that justifies the record, and access control matching the source systems.A replay path that reconstructs a run from the trace alone, used often enough that its gaps are known.

Benefits

Makes non-deterministic failures diagnosable: you can answer why this run did this, instead of reproducing until it happens again.
Turns record-keeping obligations into an artefact rather than a promise. The reconstruction either works on a random past run or it does not.
Feeds evaluation with real failures instead of invented ones, which is the difference between a benchmark and a regression suite.
Separates drift from deployment. With the model version in the record, 'it got worse' becomes a question with an answer.

Risks

The trace store as the softest target: it holds the same data as the systems it describes, usually with weaker access control and longer retention.
Capture cost that grows with traffic until someone samples on the head to save money, quietly removing exactly the runs worth keeping.
Redaction that removes what the reconstruction needed. Over-redaction is invisible until the day someone tries to replay a run and cannot.
Volume mistaken for coverage. Terabytes of spans with no shared identifier still cannot answer a single question about one run.

When not to use it

Single-step, deterministic calls where the input and the output are the whole story. A request log already reconstructs those.
Prototypes with no users and no obligations, where the cost of the trace pipeline exceeds anything you would learn from it.
Where the applicable rule forbids retaining the content at all. Then the trace records that a decision happened and its metadata, and the content stays out — that is a different artefact, and pretending otherwise creates the liability the rule was written to avoid.

Technologies

OpenTelemetry GenAI semantic conventionsDistributed tracing backendsTail-based samplingField-level redaction at captureAppend-only / WORM storageStructured logging with a propagated run id

Examples

An incident where an agent emailed the wrong customer. The run identifier links the retrieval that returned the wrong record, the tool call that used it and the message sent, so the root cause is one query rather than a week of reproduction attempts.
A regulator asks how a decision was reached six months ago. The replay path rebuilds the run from the trace alone, including the model version in force that day.
A quality drop after a model upgrade. Because every span carries the model version, the comparison is between two populations of real runs rather than between impressions.

KPIs

Reconstruction success rate: Share of randomly chosen past runs that can be rebuilt end to end from the trace alone. This is the control's own test, and it is the only number here that cannot be satisfied by capturing more.
Correlation completeness: Share of spans in a run that carry the run identifier. Anything below 100% means some step is invisible, and the missing one is rarely the boring one.
Sensitive-field escape rate: Share of sampled records containing a value the redaction rules should have removed. The target is zero; any other number means the trace store is accumulating a liability.
Anomalous-run retention: Share of errored or escalated runs retained after sampling. Head-based sampling drives this toward the sample rate, which is the failure this metric exists to catch.

Observed failure modes

The trace that proves nothing: every step is logged, no step shares an identifier, and reconstructing one run means correlating timestamps by hand.
The prompt recorded by reference. The template changed, so the log now describes a prompt the model never saw, and nobody notices until a reconstruction contradicts the output.
Head-based sampling that keeps the ordinary. The run you need was dropped at the entry point, before anything knew it was going to be interesting.
The log as the breach: raw tool arguments carry personal data into a store with broader access and longer retention than the database they came from.
Missing model version. A behaviour change and a silent model update look identical in the record, and the investigation stalls on a question the trace should have answered.

Lessons learned

Correlation is the product; capture is the raw material. Teams that buy a tracing backend and skip the identifier end up with storage rather than answers.
Test the reconstruction, not the pipeline. Pick a random past run and rebuild it — the gaps are always somewhere nobody instrumented, and only the attempt finds them.
Redact at write time. Every redaction deferred to read time is a decision to keep the raw value, and storage outlives the intention.
Sample on the tail. Head-based sampling is a decision to discard the interesting runs made before anything knows which ones those are.
Record the model version everywhere. It costs a field and it is the difference between diagnosing drift and arguing about it.

FAQs

We already use a tracing backend. Isn't this solved?: A backend gives you capture and storage. This pattern is about the three things a backend does not decide for you: whether one identifier threads the whole run, whether the fidelity is enough to reconstruct it without the source systems, and whether what you wrote down is safe to keep. Teams with excellent tooling routinely fail the reconstruction test.
Doesn't full capture conflict with data minimisation?: It would, if capture meant keeping everything raw. The control states both halves on purpose — enough to reconstruct, nothing that turns the log into the breach — and the way to hold both is field-level redaction at capture plus retention tied to the obligation that justifies the record. What you cannot do is decide the tension away by keeping everything and calling it compliance.
How much fidelity is enough?: Exactly enough to pass the reconstruction test on a run picked at random, and no more. That threshold is discoverable by trying it, which is why the test belongs in the routine rather than in an audit. Anything captured beyond it is cost and liability without a question it answers.