Empirical Evaluation of Guardrail Execution Precedence in Asynchronous Agent Runners

1. Introduction & Problem Statement

Agentic AI frameworks delegate tasks to LLMs via function calling. To maintain safety, control planes implement input guardrails intended to sanitize inputs, detect prompt injections, or enforce policy boundaries before state-modifying actions are committed.

In concurrent systems, safety enforcement requires a formal causal happens-before relation (≺) between guardrail evaluation (T₁) and side-effect commitment (T₃):

Guardrail Start (T₀) ≺ Guardrail Verdict (T₁) ≺ Tool Start (T₂) ≺ Side Effect Committed (T₃)

If chronological telemetry reveals T₃ < T₁ (side effect committed before guardrail verdict logged), the structural precedence requirement (T₁ ≺ T₃) is falsified for that execution path. Even if the framework subsequently raises a tripwire exception (T₄), the state modification (T₃) has already occurred in storage.

Expected Safe Execution (Sequential / Barrier):
  [T0: Guardrail Start] ---> [T1: Guardrail Verdict] ---> [T2: Tool Start] ---> [T3: State Commit]

Observed Parallel Execution Race Window:
  [T0: Guardrail Start] ---+---> (Async Guardrail Task) ---------------------> [T1: Verdict] ---> [T4: Tripwire]
                           |
                           +---> [T2: Tool Start] ---> [T3: State Commit]  <-- CHRONOLOGICAL RACE (T3 < T1)

2. Architectural Call-Chain Analysis (SDK v0.22.3)

Inspection of the OpenAI Agents SDK v0.22.3 source code reveals the structural root cause:

1. Runner Entry Point: Both Runner.run() and Runner.run_streamed() instantiate agents.run.AgentRunner and delegate turn execution to _run_single_turn().
2. Asynchronous Task Dispatch: When run_in_parallel=True, _run_single_turn() dispatches input guardrails as background asyncio.Task instances or via asyncio.gather().
3. Un-Awaited Tool Execution: Concurrently, assistant turn item processing forwards model function call outputs directly to tool_execution.py:execute_tool_call(). execute_tool_call() invokes the tool function without awaiting the completion of the input guardrail tasks.
4. Post-Commit Exception Raising: The runner gathers input guardrail results at turn completion. If tripwire_triggered == True, the runner raises InputGuardrailTripwireTriggered (T₄), but state modification (T₃) has already been written to the persistent sink.

3. Experimental Methodology & Harness Architecture

To isolate SDK task loop scheduling mechanics from network I/O jitter, the experiment uses:

• Offline Scripted Model: An offline ScriptedModel emitting deterministic ResponseFunctionToolCall and ResponseOutputMessage turn items.
• Persistent Local Sink: A local SQLite database (effects.db for non-streaming, stream_effects.db for streaming) recording committed_effects and telemetry_events.
• High-Resolution Monotonic Telemetry: Monotonic nanosecond timers (time.perf_counter_ns()) logging lifecycle events (T₀ to T₄).
• Latency Control: Synthetic latency parameterization: 50 ms for slow DENY guardrails, 5 ms for fast tool execution.

4. Empirical Findings Across 1,200 Trials

We executed 1,200 trials (N=100 per condition across 6 non-streaming and 6 streaming conditions):

5. Four-Way Categorization of Execution Outcomes

To maintain strict epistemological clarity, execution outcomes are partitioned into four distinct categories:

1. Category 1 — Chronological Precedence Race (T₃ < T₁): Side effect committed (T₃) before guardrail decision logged (T₁). Observed in 99.0% of parallel DENY trials (99/100 in R2-A and 99/100 in S-A).
2. Category 2 — Post-Verdict Commit Before Tripwire Exception (T₁ < T₃ < T₄): Observed in 1 trial per parallel DENY suite (R2-A Trial 31, Δ = +13.35 ms; S-A Trial 1, &Delta = +38.02 ms). Python asyncio event-loop task scheduling jitter is a plausible contributing factor for delaying T₁ logging; however, tool dispatch T₂ was still executed and committed state before T₄. The SDK runner did not block tool dispatch upon T₁ completion.
3. Category 3 — Side-Effect Persistence Prior to Tripwire Exception: In 100% of parallel DENY trials lacking an authorization barrier (100/100 in R2-A and 100/100 in S-A), local database state was committed (T₃) prior to the runner raising the tripwire exception (T₄).
4. Category 4 — Complete Side-Effect Suppression Under Authorization Barrier: Introducing an explicit application-level authorization gate (R2-D2 and S-D2) yielded 0/100 commits on DENY across 200 trials.

6. Transparency, Reproducibility & Editorial Precautions

6.1 Verification & Reproducibility Notice

All empirical statistics reported in this paper were verified locally by LatinFlash Research using frozen execution traces (r2_replication_results.json, stream_replication_results.json) and local SQLite database instances (effects.db, stream_effects.db). Researchers can independently re-run the replication harness using:

python experiments/a02-07-guardrail-happens-before/r2/r2_repro_harness.py 100 --out-dir experiments/a02-07-guardrail-happens-before/r2-repro
python experiments/a02-07-guardrail-happens-before/r2-stream/stream_repro_harness.py 100 --out-dir experiments/a02-07-guardrail-happens-before/r2-stream-repro

6.2 Status of JEV Automated Review

Evaluations generated by JEV (jev-1.13.0) represent automated LLM-based rubric assessments of report text and source excerpts. JEV scoring does not constitute external academic peer-review or independent physical validation of SQLite disk transactions. JEV results are preserved strictly as supplementary audit metadata.

6.3 Telemetry & Timezone Disclosure

Telemetry intervals (Δ = T₃ − T₁) are calculated using Python's monotonic high-resolution clock (time.perf_counter_ns()), unaffected by system clock adjustments. Wall-clock run metadata timestamps recorded in JSON headers and SQLite metadata tables are stored in standard UTC ISO-8601 format (e.g. 2026-09-24T10:56:00Z). Local workspace log timestamps correspond to CEST (UTC+2, e.g. 2026-09-24T12:56:00+02:00).

7. Methodological Scope & Boundaries

• Latency Parameterization: The 99.0% race rate is parameter-dependent (50 ms guardrail vs 5 ms tool execution). It is not asserted as a universal empirical probability for all production workloads.
• SQLite Local Sink: Local SQLite database commits demonstrate persistent state writes in an offline harness; they are not asserted as universally irreversible operations or equivalent to live external financial API transactions.
• SDK Version Scope: Conclusions apply specifically to OpenAI Agents SDK v0.22.3.

8. Recommended Mitigations

1. Proposed Framework-Level Design: We propose adding a configuration flag to RunConfig (e.g. await_guardrails_before_tool_dispatch: bool = False). When enabled, the runner would insert an explicit await barrier on all active input guardrail tasks before invoking execute_tool_call(). (Note: This is a proposed design enhancement; it is not an existing setting in SDK v0.22.3).
2. Application-Level Design Pattern: Implement explicit pre-commit authorization gates (Pattern D2) within application code, ensuring tools check input guardrail status before committing persistent side effects.

9. Provenance & Hashes

Verified Dependencies & Environment:
--------------------------------------------------------------------------------
Python Version:     3.11.15
SDK Package:        openai-agents v0.22.3
OpenAI Client:      openai v1.65.5
Pydantic:           pydantic v2.10.6
Database Driver:    sqlite3 v3.45.3
Operating System:   Windows-10-10.0.19045-SP0

Corpus SHA-256 Hashes:
--------------------------------------------------------------------------------
r1-snapshot/harness.py:
  ff5de1f3e98f06003717eaab49c0f3c6652c2d39e490960ff5d4803703885216

r2/r2_harness.py:
  25212b883bc9d79e893544c7f6fea0f4851780969de91dba330dfc701bb1d527
r2/effects.db:
  7a70f1279b827bb1749255846982a4b7fc22077a50142be0efe1055281cab95d
r2/raw_traces/r2_replication_results.json:
  a2074faabb0d4296702af301225cc16bed1b5768aab8401641c642e4937dbb04

r2-stream/stream_harness.py:
  e94e87ca2b21db97e007b7bc3f3a05e64d551d3c6579c81eee271b4d27097d45
r2-stream/stream_effects.db:
  1834285fdd2153e66275a90dfe3394204dd50913f4889b4aa2c2546e6ad4c197
r2-stream/raw_traces/stream_replication_results.json:
  16f0280c95e04ba30c9bed942a37de80da15652893ada9d0e30d4b83ad446d65
--------------------------------------------------------------------------------
Publication Manifest Verification: 57 / 57 Binary SHA-256 PASS (0 Mismatches)