v1.0.3

Controlled Replay Framework

A controlled replay is a safe, isolated new execution that changes one approved piece of tool evidence. It gives Causal Audit a trustworthy way to measure whether that evidence affected an outcome.

What controlled replay means

A controlled replay is a new execution of an already completed agent run. It begins with a frozen reference to the original run, changes only one approved piece of tool evidence, and runs that variant in isolation. The original run, its evidence, and its recorded outcome stay exactly as they were.

Think of a claims agent that checked a fraud score before recommending a manual review. A controlled replay can safely ask: “what would the measured outcome be if the approved fraud score were lower?” It does not ask the live fraud service to change a customer's record, and it does not re-send the original claim to production.

A controlled replay is visibly linked to its Causal Audit and is finalized by that audit rather than by the standard Replay comparison flow.

Not a retry or a standard replay

These three actions may sound similar, but they have different purposes and safeguards. Choosing the right one prevents accidental production work and avoids reading the wrong result.

Action	Question it answers	What it does
Retry	“Can the original request succeed now?”	The owning application tries the work again. It can cause real side effects and is outside this control plane.
Standard Replay	“How does a governed reproduction compare with its saved scorecard?”	Creates a new execution, then follows the normal evaluation, baseline comparison, and drift workflow.
Controlled Replay	“Did this approved evidence change materially affect this outcome?”	Creates an isolated variant with one governed evidence intervention and returns its bounded outcome to Causal Audit.

Why the normal Replay evaluation is not shown

A controlled replay does not run the generic Replay evaluation or drift comparison. Causal Audit already has the outcome artifact it needs to compare the original and controlled runs. Asking it for a separate historical scorecard could reject a safe causal test for the wrong reason.

Who owns each part

The framework keeps responsibilities where they belong. The platform governs whether a test is allowed and preserves its evidence; the runtime performs the isolated agent execution.

Owner	Responsibility
Runtime integration	Publishes bounded execution evidence, resolves approved evidence under its own access rules, and declares how it can run an isolated replay.
Replay adapter	The runtime-specific connector that rebuilds the source execution safely with the approved alternate evidence. It must prevent production side effects.
Intervention Policy	Defines the one allowed evidence change for a tool and schema, including its provider and version.
AI Governance Control Plane	Checks eligibility, queues work, keeps the tenant scope and lineage, scores outcomes, and persists the immutable Causal Audit result.

What an integration must provide

The platform fails closed: a controlled replay is available only when the runtime has declared the information and safety capability needed to perform it responsibly. It is better to report “not eligible” than to guess.

A successfully completed source execution in the current organization and project.
A bounded numeric outcome artifact for the original run.
One or more tool calls with a stable call ID, tool name, durable evidence reference, and evidence digest.
A schema identity and version for the tool result, so an alternate value can be checked before it is used.
An explicit replay capability: adapter name and version, frozen source reference, and supported intervention strategies.
An active, unambiguous Intervention Policy matching the selected tool and schema.

The platform can retain references, digests, policy versions, and result scores. It does not need to store raw prompts, model responses, credentials, tool arguments, tool results, or hidden reasoning traces to perform this check.

Lifecycle

The lifecycle is deliberately broken into durable steps. Each step leaves a reviewable record before the next one begins.

1. Select  → choose a completed execution and auditable tool call
2. Author  → create, preview, validate, and activate one Intervention Policy version
3. Freeze  → save the exact execution, policy, strategy, sample count, and seed
4. Create  → generate schema-valid counterfactual evidence and a controlled Replay request
5. Run     → replay adapter creates an isolated counterfactual execution
6. Verify  → Causal Audit checks lineage, compares scores, and persists its result

A policy can use NULLIFY for a neutral schema-valid value,REPLACE for an authorized alternate evidence reference, orPERTURB for a bounded field-level change. The policy authorizes the test; it does not pre-decide the answer.

An audit may request more than one sample. In that case, the framework creates one controlled replay for every required sample and calculates the counterfactual score from the complete set. The seed and sample count are saved so the request can be understood and repeated deliberately.

Workers & Recovery

The API does not wait for a replay to finish. It records a QUEUEDaudit and submits durable jobs. This makes the system resilient to a browser closing, an API restart, or a temporary worker interruption.

Causal Audit job: validate frozen inputs and create controlled Replay job(s)
Replay job:       invoke the isolated runtime adapter and save a new execution
Audit finalizer:  verify every replay, score outcomes, and complete the audit

These are two logical worker stages. They can run in the same worker service or separate worker processes, but the API and workers must share durable stores. Job leases and idempotency keys ensure that a re-delivered job reuses its durable identity instead of creating a duplicate execution.

Read Job Execution Control Plane for the job lifecycle, leases, retries, and operational controls.

Cost & Capacity

A controlled replay is real work performed by your runtime. Each requested sample can use model inference, runtime compute, tool or API calls, data services, storage, and network capacity. For example, an audit with three counterfactual samples can create three isolated replay executions for each selected tool call.

Your organization owns the cost decision

The customer or business running the integration is responsible for approving, controlling, and paying charges from its models, agent runtime, tools, data providers, cloud infrastructure, and any other connected services. AI Governance Control Plane does not set, absorb, or reimburse those external costs.

Treat Causal Audit as a deliberate review operation, not something to run automatically against every production execution. Start with a small sample count, choose a narrow evidence target, and use the runtime's own quotas, budgets, and rate limits before scaling the practice.

What is retained

A completed result is more than a label such as “Evidence Aligned.” It keeps the chain needed to answer “how did we get this result?” without retaining raw sensitive content.

active policy ID + version
  → intervention provider + digest
  → original and counterfactual evidence references + digests
  → controlled Replay ID + produced execution ID
  → original and counterfactual outcome scores
  → influence measurement + Causal Audit classification

Studio links the controlled replay back to the exact Causal Audit. Reviewers can inspect its configuration, execution, lineage, and audit outcome without recalculating the result in the browser.

When something fails

A controlled replay can fail because the policy is no longer valid, evidence cannot be resolved, an alternate value breaks the schema or policy rules, the isolated runtime fails, or an outcome cannot be scored. These are meaningful safeguards, not conditions to bypass.

No partial influence score

Every required sample must have complete policy, evidence, replay, and score lineage. If any required link fails, the audit becomes FAILED and stores safe diagnostics. It does not publish a best-effort classification or a reduced-confidence influence value.

Where to go next

Read Causal Audit for the outcome classifications and review process. Read Replay Management for the standard replay workflow, or follow Run Causal Audit to complete the operator steps in Studio and the API. Connector authors should also read Agent Runtime Integration.