The incident report is not evidence

Most teams discover their evidence model at the worst possible time: after a surprising outcome, when someone asks for an explanation and the answer is scattered across dashboards, traces, prompt histories, feature flags, and the memories of the people who were on call.

That is an incident report. It may be useful. It is not the same thing as evidence.

An incident report is reconstructed after the fact. It is selective, narrative, and optimized for communicating what a team now believes happened. Evidence is what allows that belief to be challenged. It is the durable set of identifiers, inputs, configurations, evaluations, policies, jobs, and decisions that make a claim independently inspectable.

The comfort of observability

Observability is indispensable. Metrics reveal trends, traces reveal timing, and logs reveal local context. The problem begins when teams mistake visibility for accountability.

A dashboard can show that answer quality fell last Tuesday. It usually cannot answer whether the evaluation used the same rubric as last week, whether the model alias moved, whether the policy version changed, or whether a retry produced the output being reviewed. A trace can capture a tool call, but not necessarily the governed relationship between that call and the decision it informed.

Evidence is relational

The important unit is not a giant JSON blob. It is a set of stable relationships. An evaluation belongs to an execution. A decision refers to the policy version that shaped it. An asynchronous job records who submitted it and under which request and correlation context. A replay result points to the baseline and candidate evaluations used to compare historical and reproduced behavior.

Those relationships let an operator move from a claim to the artifacts that support it without inventing the graph in the browser or in a post-incident spreadsheet. They also let the system state its limits. A missing baseline, unavailable artifact, or failed projection is visible as a missing link—not silently filled with inference.

What the evidence-fidelity work proved

We tested two related things: whether an Inspect evaluation can become durable Agent Runtime evidence, and what is lost when that ordered evidence is reduced to an ontology projection. The following claims are supported by the validation we ran.

The diary and the summary

Imagine a robot solving a treasure hunt. The robot leaves behind two records.

• Trajectory = the robot's diary: “I saw the map, opened box A, failed, tried box B, then found the treasure.”
• Projection = the final report: “The robot opened two boxes and found the treasure.”

The report preserves the outcome, but it cannot explain the robot's behaviour. That is the difference between an ordered trajectory and a useful operational summary. Both have a purpose. They are not interchangeable evidence.

A prompt variant is not a scaffold experiment

An earlier integration smoke test compared direct generation with a planning instruction added to the prompt. It made one model call in each case. Direct generation used 400 input and 80 output tokens (480 total); the instruction variant used 860 input and 239 output tokens (1,099 total) across 20 attempts: 2.29× as many tokens.

That is useful evidence that a versioned prompt configuration changes usage. It is not evidence of planner–executor overhead, because it did not make a separate planning call. The fixture is named planning_instruction_generate to make that boundary explicit.

A genuine planner–executor result

We then ran ten small, deterministic multi-step tasks twice under fixed conditions. The baseline used one answer-generation call per sample. The planner–executor candidate made a short structured plan in call one, then used that plan and the original task to answer in call two. The plan content is supplied only to that second call; governance retains its version and digest, not hidden reasoning text.

For this small controlled task set, the genuine planner–executor scaffold changed accuracy, used 4.29× the tokens, and used 1.76× the model latency. This is not a general claim that planning is better or worse. It is a reproducible observation whose model, task set, scorer, runner, limits, scaffold version, per-call usage, and fingerprint remain part of the evidence.

Why Replay needs the diary

AIGP Replay reads ordered Agent Runtime events, including actions, evidence, failures, and recovery order. Replay is therefore trajectory-grade: it works from the diary, not a summary generated later.

The ontology projection is still valuable. It preserved the final outcome, duration, and operational relationships. But the controlled recovery scenarios showed that it can lose exact ordering, retry and recovery sequence, token and score evidence, evidence-before-action relationships, and the detail needed for causal conclusions.

When only that reduced projection is available, the platform returns INSUFFICIENT_EVIDENCE. It does not invent an explanation merely because it knows the final result.

A careful boundary

The rich recovery cases in this work were controlled scenarios designed to test information loss. They are legitimate evidence that this projection can lose ordering, recovery, and causal detail. They are not a claim that every real agent run, or every agent framework, loses the same information in the same way.

The live Inspect run proved the durable lineage path. It did not establish broad tool-enabled behavioural results across models or frameworks. Those require additional representative runs with the same careful controls.

What this means for agent frameworks

Many agent frameworks are excellent at running agents. They provide planning, tools, memory, loops, and retries. Some produce rich traces too. But their event schemas and completeness guarantees vary.

A trace may show that operations happened without reliably preserving that one piece of evidence caused a decision, that a failure triggered a retry, or that a recovery produced the final outcome. Ingesting spans from LangGraph, CrewAI, AutoGen, or another framework does not automatically create governance-grade behavioural evidence.

The rule is practical: if a framework supplies only a summary or incomplete trace, AIGP limits its conclusions. If it supplies ordered trajectory events, the platform can support stronger Replay and causal analysis. This is an evidence-contract distinction, not a claim that one framework is broadly better than another.

Three controls for trustworthy review

• Reproducible evaluation: record the exact model, scaffold, runner, task set, scorer, and limits behind a result.
• Trajectory-grade Replay: preserve the ordered evidence needed to understand behaviour.
• Honest uncertainty: refuse causal claims when only a lossy projection is available.

Other systems may report that an agent succeeded or failed. A governance control plane should also determine whether enough evidence exists to explain how and why—and say plainly when it does not.

Audit is not a graveyard

Audit systems often become write-only graveyards: lots of events, little ability to answer a question. A useful audit record needs a job or resource identity, an operation, actor and tenant context, timestamps, outcome, and enough correlation metadata to connect it to the governed work.

That does not mean storing every secret, prompt, or raw artifact in every audit event. In fact, duplicating sensitive data widely is a poor governance strategy. It means recording safe, durable references and enforcing a clear boundary around where fuller evidence lives.

The test

There is a simple test for whether a system has evidence rather than merely logs. Give an operator a decision ID, a job ID, or an execution ID and ask them to answer these questions without opening a terminal or asking the original developer:

• What was requested, by whom, and in which tenant/project?
• What configuration, policy, and evaluator versions were in force?
• What execution and evaluation artifacts support the outcome?
• Which evidence was compared, and what changed?
• What is known, and what is explicitly unavailable?

If the answer requires joining five systems by timestamp and intuition, the operational architecture is carrying more trust than it can justify.

Make review native

AIGP treats review as a first-class workload. Jobs are durable; identities are idempotent; evaluation history is retained; replay lineage and comparison references are persisted; Studio renders backend-owned read models rather than manufacturing governance conclusions in React.

The deeper point is broader than any one implementation. Systems that make decisions should be built to receive questions. Not just “is it up?” or “what is the error rate?” but “why did this outcome deserve to be trusted?”