Failure Semantics, Cancellation, and Side-Effect Idempotency in AI Agent Runners

Abstract & Central Claim

When AI agents execute tool-based operations, failure handling and cancellation semantics determine whether external systems suffer duplicate side-effects, orphaned transactions, or corrupted state. This paper evaluates the failure semantics and recovery behavior of AI agent runners under twelve operational fault conditions (F₁ to F₁₂), comparing an un-governed runner baseline against a transactional governed architecture.

Using an independent process-level HTTP service backed by append-only WAL-mode SQLite audit databases, we conducted a multi-stage empirical audit comprising N = 366 total evaluated trials across an Acceptance Pilot (N = 78), a Main Replication (N = 234), and a Real-Fault Complementary Suite (N = 54).

Central Claim:
In un-governed AI agent runners subjected to concurrent request races (Fault F₉), side-effect execution defaults to non-deduplicated dispatch, yielding 48 additional un-governed duplicate effects across the acceptance pilot (12) and main replication (36). Across the 153 ungoverned durable business effects recorded in this constructed experimental corpus, 48 additional duplicate records were committed. This 48/153 ratio reflects the composition of observed effects within this specific trial suite and does not represent a general failure probability.

Conversely, under the governed architecture - which incorporates client-side idempotency keys, pre-commit authorization barriers, and WAL-backed state reconcilers - zero governed duplicate effects were observed (0 duplicates across 105 governed durable business outcomes) in the tested conditions. Governed failure semantics enforce at-most-once durable side-effects, idempotent replay, and deterministic crash recovery across all audited SDK releases (v0.20.0, v0.21.1, v0.22.3).

1. Introduction & Problem Statement

Agentic orchestration systems delegate multi-step operational logic to LLMs while invoking external HTTP APIs, databases, and financial microservices. In production environments, execution loops are subject to operational failures including network transport timeouts, process crashes, user cancellations, and concurrent request races.

Without formal transactional boundary controls at Layer 6 (Resilience & Recovery) of the Vertical AI Governance Stack, failure recovery defaults to un-synchronized retry loops. Audit A02-09 formalizes a twelve-part failure taxonomy (F₁ to F₁₂) and empirically quantifies state consistency, duplicate side-effect occurrences, and crash recovery behavior under governed vs. un-governed runner architectures.

2. Failure Taxonomy & Verified Mechanisms (F₀ to F₁₂)

Fault ID	Scenario Name	Injection Point & Verified Mechanism	Ungov Effects	Gov Effects	Ungov Duplicates	Client Observation & Recovery Status
F0	Baseline Control	Normal execution (no fault injected)	1	1	0	Both: HTTP 200 OK
F1	Connection Refused	Socket connection refused (ConnectError)	0	0	0	Both: 0 commits, safe retry
F2	Pre-Commit Rejection	Server handler HTTP 400 validation error	0	0	0	Both: 0 commits, HTTP 400 Bad Request
F3	Pre-Commit Timeout	Socket ReadTimeout (5.0s delay vs 1.5s client timeout)	0	0	0	Ungov: UNKNOWN. Gov: NOT_COMMITTED_RECOVERED
F4	Post-Commit Drop	Socket drop after DB commit	1	1	0	Ungov: UNKNOWN. Gov: COMMITTED_RECOVERED
F5	Pre-Commit Cancel	Client cancel trigger during pre-commit loop	0	0	0	Ungov: CancelledError. Gov: NOT_COMMITTED_RECOVERED
F6	Post-Commit Cancel	Client cancel trigger during post-commit parsing	1	1	0	Ungov: UNKNOWN (phantom commit). Gov: COMMITTED_RECOVERED
F7	Idempotent Retry	Identical payload retry on idempotency engine	1	1	0	Gov: HTTP 200 OK cached replay hit (1 commit)
F8	Conflicting Retry	Payload mismatch on idempotency engine	1	1	0	Gov: HTTP 409 Conflict payload mismatch (1 commit)
F9	Concurrent Retries	5 concurrent HTTP requests dispatched per trial	5	1	4	Ungov: 5 commits per trial (4 duplicates). Gov: 1 commit per trial (0 duplicates).
F10	Pre-Commit Crash	Server hard-terminated via os._exit(101) before commit	0	0	0	Ungov: UNKNOWN. Gov: NOT_COMMITTED_RECOVERED (verified absence of commit)
F11	Post-Commit Crash	Server hard-terminated via os._exit(102) after WAL commit	1	1	0	Ungov: UNKNOWN (connection drop, 1 commit). Gov: COMMITTED_RECOVERED
F12	Process Recovery	Cross-process worker restart / reconciliation query	1	1	0	Ungov: UNKNOWN. Gov: COMMITTED_RECOVERED

3. Experimental Corpus Accounting (N = 366 Total Scope)

Experimental Suite	Executed Trials	Ungoverned Durable Effects	Governed Durable Effects	Ungoverned Duplicates (F9)	Database & Traces
Acceptance Pilot	78	36	24	12	pilot.db (78 JSON traces)
Main Replication	234	108	72	36	main.db (234 JSON traces)
Real-Fault Suite	54	9	9	0	complementary.db (54 JSON traces)
Consolidated Total	366	153	105	48	3 SQLite DBs (366 Traces)

Exact Denominators: 366 executed trials across 3 WAL-mode SQLite DBs, yielding 258 durable business effects (153 ungoverned, 105 governed), 48 additional ungoverned duplicate effects observed strictly in condition F9, and 0 governed duplicate effects across all conditions.

4. Empirical Findings & Recovery Semantics

4.1 Abrupt Process Termination (F₁₀ vs F₁₁)

A critical distinction exists between early experimental iterations and the final real-fault complementary suite:

Simulated Fault Phase: Initial pilot iterations simulated F₁₀ and F₁₁ using software exception flags within the runner event loop.
Real-Process Crash Phase: The final complementary suite (N = 54) executed hard OS process kills using native os._exit(101) (pre-commit) and os._exit(102) (post-commit) calls to test out-of-process state recovery without software mocks.

Observed Real-Crash Recovery Outcomes (N = 54 Suite):

Pre-Commit Abrupt Crash (F₁₀ - os._exit(101)): Ungoverned runners yield an unresolved UNKNOWN status (0 DB records committed). Governed runners query the DB, confirm a verified absence of commit (0 DB records found), and classify state as NOT_COMMITTED_RECOVERED (9/9 trials). No database modification is required because no rows were written prior to process kill.
Post-Commit Abrupt Crash (F₁₁ - os._exit(102)): Ungoverned runners yield UNKNOWN status (1 DB record committed prior to crash, but client receives connection drop). Because no retry logic was executed in the test, the outcome remains UNKNOWN without subsequent duplicate creation. Governed runners inspect server WAL logs, identify committed transactions, and recover state to COMMITTED_RECOVERED (9/9 trials) without duplicate re-execution.

4.2 Concurrency & Idempotency Races (F₉)

Under condition F₉ (5 concurrent requests per trial), the ungoverned runner lacks request deduplication barriers. Each of the 5 concurrent requests commits 1 durable business effect, yielding 5 total durable effects per ungoverned trial (1 primary + 4 additional duplicates). Across 3 pilot trials (12 duplicates) and 9 main replication trials (36 duplicates), a total of 48 additional duplicate effects were committed. The governed runner's idempotency key registry intercepts concurrent arrivals, returning existing committed results (4 idempotent reuses per trial, 0 duplicate DB records).

5. TypeSafe AI JEV External Review Scope & Results

The submitted report text snapshot was evaluated by TypeSafe AI's System One model engine (jev-1.13.0) via an authentic authenticated HTTP 200 OK request (https://api.typesafe.ai/v1/systemone).

Dimension	Score (0–3)	Confidence	Evaluator Model Interpretation
falsifiability	2.99	0.99	Highly operationalized, clear metrics, fully falsifiable hypotheses
evidence_alignment	2.88	0.88	Claims very strongly supported by presented trial data
stack_consistency	2.51	0.51	Terminology strictly aligned with L6 Resilience & Recovery spec
methodology_clarity	1.55	0.39	Clear to partial separation of hypothesis vs interpretation
reproducibility	1.41	0.49	Partially reproducible (noting dependency on local DBs & harnesses)
claim_overreach	0.68	0.46	Claims consistently bounded by presented empirical evidence

Critical Review Scope Boundary Notice:
1. Evaluated Input: TypeSafe AI System One (jev-1.13.0) evaluated ONLY the raw text snapshot of the submitted report (state) and six scoring questions.
2. Unexamined Artifacts: The JEV API did NOT directly inspect the 366 raw trace JSON files, the three SQLite databases, the Python harness/service source code, or the evidence manifests.
3. Nature of Evaluation: The JEV response represents an automated model-based structural review of claim-boundedness, evidence alignment, and textual consistency. It is NOT human peer review, mathematical proof, or independent empirical replication.

6. Scope Boundaries & Methodological Limitations

Deterministic Scripted Models: Tool invocation decisions were generated by an offline deterministic adapter (ScriptedModelAdapter) to isolate runner mechanics from LLM non-determinism.
Localhost & SQLite Environment: Network interactions were conducted over 127.0.0.1 and state was stored in local SQLite databases in WAL mode. Real-world WAN latency, packet loss, and multi-region network partitions were not evaluated.
Historical SDK Version Bounds: Empirical observations apply specifically to OpenAI Agents SDK releases v0.20.0, v0.21.1, and v0.22.3.
Per-Condition Sample Sizes: Evaluated sample sizes (9 or 18 trials per condition per suite) are designed for structural fault auditing rather than high-volume statistical probability estimation.
Specific Duplication Context: Duplicate side-effects were observed specifically under condition F₉ (concurrent request race). They should not be extrapolated as a universal duplication rate across unrelated failure modes.
Controlled Fault Injection: Faults F₁ to F₈ were injected via controlled test harness hooks rather than unannounced hardware failures.

7. Recommended Governance Mitigations (L6 Layer)

Client-Side Idempotency Key Injection: Assign unique deterministic idempotency keys to outgoing tool side-effect payloads prior to transport dispatch.
Pre-Commit Authorization Barriers: Require agent runners to log intent in a local WAL database before issuing network requests.
Out-of-Process State Reconcilers: Deploy independent recovery daemons to inspect pending transaction logs following process crashes (F₁₀, F₁₁).

8. Methodology, Reproducibility & Data Availability

Execution environment: Windows 11 AMD64, Python 3.11.15, OpenAI Agents SDK v0.20.0, v0.21.1, v0.22.3.

The complete release candidate package a02-09-final-editorial-candidate.zip contains all 3 SQLite databases, 366 trace files with SHA-256 digests, test harnesses, authentic JEV transaction records, and consolidated evidence manifest A02-09_EVIDENCE_MANIFEST.json.