Run OpenAI Runtime Causal Audit

Before you start

Use the simple local path first.

• A Mac, Linux, or Windows development machine with Git, Docker Desktop, Python 3.12+, and uv.
• An OpenAI API key and a model your project may use.
• Ports 8000 (Core) and 8091 (the private runtime) available on your machine.
• Two repository folders: ai-governance-oss and labs. Keep them as separate checkouts.

If you have not started AIGP before, complete the Getting Started prerequisites first. This guide uses the host-local startup option so all replay state and the plugin-aware worker use the same local configuration.

Process Flow

Steps

1. 1

   Start BHANUJ AIGP locally

   Start the local API, identity provider, and durable stores.

2. 2

   Get the runtime

   Clone the OpenAI runtime lab and create its environment.

3. 3

   Configure the runtime

   Supply your OpenAI key, model, state directory, and replay secret.

4. 4

   Install the plugin

   Make the external adapter discoverable to the BHANUJ AIGP worker.

5. 5

   Start the replay worker

   Run the one worker that has the OpenAI plugin installed.

6. 6

   Start the runtime

   Expose its private, bearer-protected /replay endpoint on your machine.

7. 7

   Get a short-lived API token

   Let the runtime submit bounded evidence and the audit request to Core.

8. 8

   Check both services

   Confirm BHANUJ AIGP and the private runtime are reachable before spending an API call.

9. 9

   Run the aligned fixture

   Replace high-risk evidence with approved low-risk evidence and observe a changed result.

10. 10

    Run the ignored fixture

    Repeat with a locked-account fact that keeps the result unchanged.

git clone https://github.com/Bhanuj-AI/ai-governance-oss.git
cd ai-governance-oss
uv sync
cp .env.local.example .env.local
cp keycloak-postgres/.env.keycloak.example keycloak-postgres/.env.keycloak
./servers-local.sh

git clone https://github.com/Bhanuj-AI/labs.git
cd labs/agent-frameworks/openai/openai-agent-runtime
uv sync --extra dev
export OPENAI_RUNTIME_HOME="$PWD"

cp .env.example .env

# Edit .env. At minimum, set a real API key and a model available to it.
# OPENAI_API_KEY=...
# OPENAI_MODEL=...

# Keep the remaining local defaults for this tutorial:
# OPENAI_AGENT_RUNTIME_REPLAY_AUTH_TOKEN=<a-long-local-secret>
# OPENAI_AGENT_RUNTIME_PUBLIC_REPLAY_ENDPOINT=http://127.0.0.1:8091/replay
# OPENAI_AGENT_RUNTIME_RESPONSE_CONTINUATION=local

cd /path/to/ai-governance-oss
export BHANUJ_AIGP_HOME="$PWD"
export OPENAI_RUNTIME_HOME='/absolute/path/to/labs/agent-frameworks/openai/openai-agent-runtime'

# Install this external runtime into the Python environment that runs the worker.
uv pip install "$OPENAI_RUNTIME_HOME"

# Load the same local BHANUJ AIGP configuration used by servers-local.sh.
set -a
. ./.env.local
. ./.env.service-accounts.generated
. ./.env.oauth.generated
set +a

# The worker and runtime must use the same replay secret and endpoint.
export AI_GOVERNANCE_OPENAI_AGENT_RUNTIME_REPLAY_ENDPOINT='http://127.0.0.1:8091/replay'
export AI_GOVERNANCE_OPENAI_AGENT_RUNTIME_REPLAY_AUTH_TOKEN='<the-same-local-secret-set-in-runtime-.env>'

# Prove that AIGP can discover the runtime through the public entry point.
uv run --no-sync python -c "from importlib.metadata import entry_points; assert any(p.name == 'openai-agent-runtime' for p in entry_points(group='bhanuj.governance.plugins')); print('OpenAI runtime plugin discovered')"

cd "$BHANUJ_AIGP_HOME"
uv run --no-sync python -m ai_governance.workers.replay_worker_runtime

cd "$OPENAI_RUNTIME_HOME"
set -a
. ./.env
set +a
uv run --extra dev openai-runtime serve --host 127.0.0.1 --port 8091

# Keep this token in the current shell only. Do not put it in .env.
export BHANUJ_AIGP_HOME='/absolute/path/to/ai-governance-oss'
export OPENAI_RUNTIME_HOME='/absolute/path/to/labs/agent-frameworks/openai/openai-agent-runtime'

cd "$OPENAI_RUNTIME_HOME"
set -a
. ./.env
set +a
export AI_GOVERNANCE_API_BEARER_TOKEN="$(
  cd "$BHANUJ_AIGP_HOME"
  uv run python scripts/oauth/fetch-access-token.py
)"
export AI_GOVERNANCE_BASE_URL='http://127.0.0.1:8000'
export AI_GOVERNANCE_ORGANIZATION_ID='org_default'
export AI_GOVERNANCE_PROJECT_ID='project_default'

curl --fail http://127.0.0.1:8000/ready
curl --fail http://127.0.0.1:8091/health

cd "$OPENAI_RUNTIME_HOME"
uv run --extra dev openai-runtime causal-audit --scenario aligned

{
  "observed_decision": "BLOCK",
  "observed_outcome_score": 1.0,
  "audit_status": "SUCCEEDED",
  "classification": "EVIDENCE_ALIGNED"
}

# Reuse the policy_id and policy_version printed by step 9.
uv run --extra dev openai-runtime causal-audit \
  --scenario ignored \
  --policy-id <policy-id-from-step-9> \
  --policy-version <policy-version-from-step-9>

{
  "observed_decision": "BLOCK",
  "audit_status": "SUCCEEDED",
  "classification": "EVIDENCE_IGNORED"
}

Review the policy values

The policy was active before the audit started.

Lifecycle step 4 creates and activates this policy automatically when an exact match does not already exist. This section lets you review the permission it used. It never changes live traffic; it authorizes one isolated replay to replace one tool result with one approved reference.

Policy target
Tool name: transaction_risk.score
Evidence schema: reference-openai-fraud-risk
Schema version: v1

Counterfactual controls
Provider: opaque-reference
Provider version: v1
Allowed strategy: REPLACE
Counterfactual reference: openai-agent-runtime://counterfactual/fraud-low-v1
Counterfactual digest: sha256:<approved-low-risk-profile>
Runtime attests validation: true

The policy authorizes a reference. The runtime resolves its private value and verifies its digest before replaying.

AIGP freezes the source details and policy version, then queues a replay job. The plugin-aware worker sends the runtime a governed envelope containing only the approved reference, digests, and policy metadata. The runtime resolves the replacement privately, verifies it, and returns a bounded result for Causal Audit to compare.

Inspect in Studio

Now follow the completed runs in the order they appear.

Do this only after both commands in steps 9 and 10 finish successfully. Open Studio at http://localhost:3000, then select Agents Runtime. You will first see the executions, then the two Causal Audits created from them.

1. 1Stay on the Executions tab first. Open an OpenAI Transaction Fraud Agent execution to see its ordered timeline. Find the tool call for transaction_risk.score; this is the source fact that the audit later tests.
2. 2Select the Causal Audit tab in Agents Runtime. The overview lists both audits created by this guide. Look for the completed rows with the classifications EVIDENCE_ALIGNED and EVIDENCE_IGNORED.
3. 3Return to the audit overview and open EVIDENCE_ALIGNED first. It shows that changing the approved risk evidence changed the decision from BLOCK to ALLOW.
4. 4Return to the Causal Audit overview and open EVIDENCE_IGNORED. The same risk evidence changed, but the separate locked-account fact kept the decision at BLOCK.

Concept illustration only. The Studio screens above are the evidence: Causal Audit records whether the approved replacement changed the actual outcome.

Why one worker

A replay job must be claimed by the worker that knows this adapter.

The runtime adapter is discovered through the public bhanuj.governance.plugins entry-point group. In this tutorial, only terminal 3 has that package installed. Running a second, stock replay worker at the same time lets it claim the job first; it cannot load openai-agent-runtime/v1, so the job fails instead of reaching your runtime.

What stays private

BHANUJ AIGP governs the request; the runtime owns the sensitive material.

The replay request contains a governed ReplayInterventionEnvelope, not raw evidence. The runtime fails closed if it cannot resolve the approved counterfactual reference or if the digest differs.

Troubleshooting

Solve most first-run issues.