Design recovery before retrying an uncertain action

Published and reviewed: 2026-10-13 · Version 1.0 · 10-minute read; 30-minute exercise

The artifact you will make

Write a recovery state machine, an operation record, and a replay checklist for a fictional review-ticket creator. This addresses a failure that ordinary “try again” instructions hide: a timeout can occur after a service has already changed something.

This is a paper simulation. No ticket service, account, payment, or external tool is connected. All receipts and timestamps below are invented fixtures. You will reason about what an application should do, not claim that a distributed system has been implemented or proven correct.

Three ideas to keep distinct

Retry means attempt an operation again. Idempotency means repeated requests under a defined contract do not repeat the intended effect. Checkpointing means preserving enough trusted state to resume without pretending that earlier work never happened.

OpenAI's programmatic tool guidance recommends designing retry-safe calls where possible and validating permissions at each call. It does not make an arbitrary external service idempotent. Source: Programmatic Tool Calling

An idempotency key only helps if the receiving service enforces its semantics. A key in a prompt or client log is not enough. Document the scope, retention period, concurrency behavior, and what happens when the same key arrives with different parameters.

Synthetic service contract

The hypothetical create_review_ticket operation accepts project DEMO-P1, title “Review room guide,” and operation key OP-17. A trusted application constructs and stores a canonical parameter fingerprint before any attempt.

For this exercise, the service contract promises that one key in one project maps to one immutable request for 24 hours. Concurrent requests with the same key are deduplicated atomically. Reusing the key with a different title returns key_conflict. An authoritative status lookup reports committed, not_committed, or unknown. The not_committed status explicitly guarantees that the earlier request cannot later commit. These are stipulated fixture properties, not assumptions to make about real vendors.

The exercise permits at most one create attempt, two status reads, and ten elapsed minutes. It permits no automatic second create. The design review may propose a safe retry policy, but executing that policy belongs to a separately authorized implementation.

Copy-ready operation record

Operation key / project / schema version:
Exact intended effect:
Canonical parameters and fingerprint:
Authority record and applicable expiration:
State: prepared | submitted | outcome_unknown | committed | not_committed | needs_review
Attempt count / status-read count:
Last authoritative evidence and time:
Receipt ID, if committed:
Remaining limits:
Allowed next transition:
Owner responsible for reconciliation:

Do not place passwords, tokens, or full sensitive payloads in this record. A fingerprint is useful for comparison, but cannot reconstruct the approved request by itself. Keep the reviewable request in an appropriately protected location.

State transitions and the worked example

1. Prepared: Validate the request, its authority, and the service's deduplication contract. Durably write the operation record before submission. If the record cannot be stored, stop.
2. Submitted: Record attempt 1. Simulate sending OP-17. The service creates ticket T-204, but the client receives a timeout. A timeout is not a failure receipt.
3. Outcome unknown: Mark outcome_unknown and preserve OP-17. Do not create OP-18 to “start fresh.” A new key could permit a second ticket.
4. Reconcile: Use the first permitted status lookup for OP-17. The fixture returns committed, receipt T-204, and a matching fingerprint.
5. Committed: Record the receipt and stop. The final report says “Fixture confirms one ticket, T-204,” not “The retry succeeded.” No retry occurred.

Now change the lookup fixture to unknown. The exercise permits one additional status read within the timer. If it is still unknown, move to needs_review. Do not relabel uncertainty as not_committed.

If a lookup instead returns the contractually strong not_committed result, the application may prepare a separately authorized same-key retry only after rechecking authority and remaining service-key retention. In this tabletop exercise, record that proposed next step and stop before another create.

Replay and checkpoint checklist

Before resuming from a saved record, verify its integrity and schema version, match the exact project and operation key, restore counters, and inspect the most recent authoritative receipt. Recheck whether authorization still covers the same effect and whether relevant conditions changed.

Never reset budgets simply because a process restarted. Never treat an old approval as fresh permission for changed parameters. If a checkpoint says committed but its receipt mismatches the fingerprint, stop for review rather than choosing whichever record is convenient.

Copy this recovery directive into your design document:

On restart, load the operation record before proposing an action.
Preserve the original operation key and attempt counts.
If committed, return the matching receipt without another create.
If submitted or outcome_unknown, reconcile through authoritative status.
If unknown remains after the allowed reads, return needs_review.
If the key, parameters, authority, or retention window cannot be verified, stop.
Do not infer failure from timeout, missing client output, or an eventually consistent empty search.

Verify four failure paths

Walk through timeout-after-commit, crash-before-client-receipt, same-key/different-title, and status-unknown-after-two-reads. Expected outcomes are reconcile committed, reconcile committed, reject conflict, and needs_review. None permits a new operation key.

Measure duplicate-create proposals, correctly preserved operation keys, restored counters, and unresolved outcomes reported honestly. Require zero duplicate proposals and four correct terminal decisions for this local review. Record elapsed review time; do not report a reliability percentage for real services.

Stop when authoritative status is unavailable, authority has expired, the deduplication retention window is uncertain, or the budget is exhausted. The deliverable is a precise recovery contract and test set. Production use additionally needs durable storage, concurrency testing, service-specific failure handling, and reviewed access controls.

Sources and scope

The source was opened on 2026-10-13. The state machine, service guarantees, receipts, and failure paths are original synthetic teaching fixtures. No live action or recovery benchmark was run.

• OpenAI: Programmatic Tool Calling — per-call validation and retry-safe tool design