Job Execution Control Plane

Job Lifecycle

Every job moves through an explicit state machine. Transitions are recorded as audit events, so the full history of a job is always recoverable.

• QUEUED — accepted and waiting for a worker.
• RUNNING — claimed by a worker and executing.
• SUCCEEDED — completed and results committed.
• FAILED — terminated after exhausting retries.
• CANCELLED — explicitly cancelled by an operator or policy.

Idempotent Submission

Submissions carry an idempotency key. Re-submitting the same key returns the existing job rather than creating a duplicate, which makes client retries and at-least-once delivery safe.

POST /api/v1/jobs
Idempotency-Key: submit-execution-2048

{
  "type": "evaluation",
  "payload": { "dataset_id": "support-agent", "policy_id": "production-ai-policy" }
}

// 200 OK (returns the same job on repeat)
{ "job_id": "job-5521", "status": "QUEUED" }

Retries

Failed jobs are retried according to a configurable backoff policy. Each attempt is recorded with its error, so operators can distinguish a transient failure from a systematic one. Non-retryable errors move a job directly to FAILED.

Worker Lifecycle

Workers claim queued jobs using a lease. A worker must renew its lease while running; if it stops renewing — because it crashed or was rescheduled — the lease expires and the job becomes eligible for another worker. This prevents jobs from being lost when a worker dies mid-flight.

Worker Health

A worker registers a heartbeat while it polls, so Studio can distinguish a healthy worker from one that has not started or has stopped reporting. This is separate from the lease heartbeat on an individual running job.

Set AI_GOVERNANCE_WORKER_HEARTBEAT_STALE_SECONDS above the polling interval to avoid normal scheduling jitter appearing as a failure. Give every worker a unique worker ID. Stale records after a crash are retained for diagnosis rather than silently deleted.

Replay Worker Dispatch

The standard worker runtime consumes REPLAY_EXECUTION, REPLAY_EVALUATION, EVALUATION, and EXPERIMENT, and CAUSAL_AUDIT jobs. A replay execution commits its produced execution identity before it queues evaluation, so worker redelivery cannot create another Replay or replay execution.

Run the worker independently in local development, or set AI_GOVERNANCE_RUN_REPLAY_WORKER=true to run it beside the API:

uv run python -m ai_governance.workers.replay_worker_runtime

Cancellation is checked at safe handler boundaries. If a process stops, its lease expires and eligible work can be claimed again without duplicating durable identities.

Causal Audit job flow

A Causal Audit uses the same durable job framework as Replay, but it has a different finish line. A normal replay creates a new evaluation, comparison, and drift result. A controlled causal replay instead returns a bounded outcome artifact to the Causal Audit worker, which calculates evidence influence.

CAUSAL_AUDIT job
  → validate frozen audit inputs and create controlled Replay job(s)
  → REPLAY_EXECUTION job calls the isolated runtime adapter
  → save counterfactual execution and bounded outcome artifact
  → CAUSAL_AUDIT finalization job calculates influence and classification

This hand-off gives each worker one responsibility. The audit handler governs the policy, evidence lineage, sample set, and final classification. The replay handler governs the isolated execution itself. Both commit their durable state before the next job is scheduled, so a process restart can resume safely rather than losing or duplicating a replay.

All required samples must complete with matching policy, evidence, replay, and score lineage. If evidence resolution, intervention generation, a replay, or scoring fails for one required sample, the audit becomes FAILED; the platform never publishes a best-effort influence value. Read Causal Audit for eligibility and reviewer-facing result meaning, or Controlled Replay Framework for the end-to-end integration and lineage model.

Persistence

Job records, attempts and state transitions are stored in the relational backend. Because state is durable, the control plane can restart at any time and resume tracking in-flight work without loss.

Operational Controls

• Cancel a queued or running job.
• Pause and resume a job queue by type.
• Inspect attempts, leases and errors for any job.
• Requeue interrupted jobs after investigation.