What you will prove
Prove that a tool result changed an agent decision.
A trace shows that an agent called a tool. Causal Audit answers a stronger question: did that tool result affect the decision? It approves one small change, asks the runtime to run again with its saved private evidence, and compares the two outcomes. The original run is never changed.
You will run
The Transaction Fraud Causal Guide Agent. Its profile and device checks run in parallel, then feed transaction_risk.score.
You will confirm
A small confidence change moves the result from REVIEW 0.90 to ALLOW 0.25. The final finding is EVIDENCE_ALIGNED.
How it works
The runtime keeps the data. Core keeps the audit trail.
Private runtime
Synthetic Agent Runtime
Runs the tools, keeps their private data, applies the approved change, and runs the decision again.
Governance record
AIGP
Approves the change, records what happened, and classifies the result.
Prerequisites
What you need.
- Python 3.12 or later, Git, and Docker Desktop.
- A running AIGP stack with the API, replay worker, and an operator account for the project.
- Local OAuth settings so the runtime can send its normal execution evidence.
- One separate secret that protects the runtime replay endpoint.
/replay. Put the same secret in both replay variables below. Do not commit it.git clone https://github.com/Bhanuj-AI/synthetic-agent-runtime.git
cd synthetic-agent-runtime
python3.12 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install -e '.[dev]'# Use one locally generated secret in both processes. Do not commit it.
export REPLAY_SHARED_SECRET='replace-with-a-local-secret'
# Terminal 1 — Synthetic Agent Runtime
export SYNTHETIC_RUNTIME_REPLAY_AUTH_TOKEN="$REPLAY_SHARED_SECRET"
# Terminal 2 — AI Governance replay worker
export AI_GOVERNANCE_SYNTHETIC_RUNTIME_REPLAY_AUTH_TOKEN="$REPLAY_SHARED_SECRET"
export AI_GOVERNANCE_SYNTHETIC_RUNTIME_REPLAY_ENDPOINT='http://host.docker.internal:8090/replay'Run the fraud flow
Create the source run.
Start the runtime, then run its built-in fraud scenario. Core receives safe operational facts. The runtime keeps tool inputs and outputs private.
.venv/bin/python -m synthetic_agent_runtime serve \
--host 127.0.0.1 \
--port 8090.venv/bin/python -m synthetic_agent_runtime run \
--scenario fraud-causal-guide \
--seed 42 \
--duration 1Synthetic Agent Runtime run: <run-id>
External execution: sar-<run-prefix>-0000000
Core execution: <platform-execution-id>
Source decision: REVIEW
Source outcome score: 0.90Inspect fan-in
Check the inputs before changing anything.
Open the source run in Agents Runtime. The profile and device calls share a tool_call_group_id, so they run independently. The risk score uses both. Claim history runs later and depends on the risk score.
- 1customer_profile.lookupgroup: decision-inputs · no dependencies
- 2device_risk.lookupgroup: decision-inputs · no dependencies
- 3transaction_risk.scoredepends on: customer_profile.lookup, device_risk.lookup
- 4claim_history.lookuplater observation · depends on: transaction_risk.score
Configure policy and audit
Approve one small change.
In Studio, open Agents Runtime, choose the Causal Audit tab, then open Intervention Policies. Create and activate the policy before creating the audit. The policy says exactly which tool result a replay may change and what change is allowed.
- 1Click Create Policy. Under Select Observed Tool Evidence, choose the completed fraud run printed by the runtime.
- 2Choose
transaction_risk.score. Let Studio fill in its tool name and schema details. Do not type them by hand. - 3Choose the structured JSON provider and PERTURB. Enter the control below, then validate and activate policy version 1.
- 4Return to Causal Audit → Audits. Create an audit for the same run, select the active policy, and submit it. Keep the source execution ID for the replay step.
Policy target
Tool name: transaction_risk.score
Evidence schema: synthetic-fraud-risk
Schema version: v1
Counterfactual controls
Provider: structured-json
Provider version: v1
Strategy: PERTURB
Field path: /confidence
Operation: NUMERIC_DELTA
Delta: -0.50
Permitted range: 0.00 to 1.00
Samples: 1Intervention: original evidence → controlled PERTURB → replay result.
Why these policy values?
- Exact tool and schema
- The run has several tool calls. Only
transaction_risk.scoreuses both parallel inputs and produces the confidence used in the decision. Exact matching stops this policy from changingcustomer_profile.lookup,device_risk.lookup, or later evidence. - PERTURB, not REPLACE or NULLIFY
- The question is simple: does a small reduction in risk confidence change the decision? PERTURB tests that without inventing new private data or removing the evidence.
- /confidence, −0.50
- The observed confidence is 0.90. Subtracting 0.50 gives 0.40. That is the approved input for this tutorial replay.
- 0.00–1.00 range and one sample
- Confidence must stay between 0.00 and 1.00. One sample is enough because this tutorial is deterministic. Use a new policy version for wider robustness testing.
Core freezes the source details and submits a replay job. The worker calls the runtime with its replay secret. Before it changes anything, the runtime checks the execution ID, tool-call ID, evidence digest, policy, approved change, and saved snapshot.
Keep evidence private
A digest proves a value without sharing it.
Core receives
- Execution and tool-call IDs
- Safe evidence references and digests
- Policy identity and outcome score
The runtime keeps
- Tool inputs, outputs, and saved evidence
- The source outcome and replacement evidence
- The change it applies and its digest
source_evidence_digest identifies the original evidence. intervention_provenance_digest identifies the approved change. After replay, counterfactual_evidence_digest identifies what the runtime produced. Core does not receive the private evidence itself.
Find the replay and result
Check the replay, then read the audit result.
Open Replay from the left navigation. In Replay Management, paste the Core execution ID from the source run into Source execution ID. Open the controlled causal replay whose source is agent-runtime-source:<your-core-execution-id>, then choose Details.
- 1On the replay overview, check for EXECUTION COMPLETED and Controlled Causal Replay. This confirms you opened the causal replay, not an older demo or history row.
- 2Open the Audit tab. You should see the source evidence frozen, the replay job submitted, and the replay succeed.
- 3Choose View evidence influence, or open Agents Runtime → Causal Audit → Audits and select this audit. Choose Explain to see the result.
Counterfactual decision: ALLOW
Counterfactual outcome score: 0.25
Counterfactual evidence digest: sha256:<runtime-computed-digest>
Causal classification: EVIDENCE_ALIGNED
Evidence influence: +0.650Result: REVIEW 0.90 → ALLOW 0.25 · influence 0.65 · EVIDENCE_ALIGNED.
Continue with the runtime source.
Use the Synthetic Agent Runtime repository for setup details, scenario code, and regression tests.
Related platform docs
- Causal Audit — classifications, evidence influence, and audit outcomes.
- Agents Runtime — observed execution evidence and tool-call context.
- Intervention Policies — scoped, versioned counterfactual controls.
- Replay Management — locating, inspecting, and auditing replay work.
- Controlled Replay — the governed replay lifecycle and boundary.
