# Make a cost, latency, and quality decision from a run log

Published and reviewed: 2026-10-13 · Version 1.0 · 9-minute read; 30-minute exercise

## The artifact you will make

Create a compact run ledger and a keep, revise, or stop decision record. Use it when someone says an agent is “faster” or “cheaper” and you need to know what was measured, which failures were included, and whether the comparison preserved the task's quality requirements.

This guide supplies invented numbers for arithmetic practice. They are not model benchmarks, current API prices, or observed improvements. No paid calls or agent deployment are needed. You can complete the exercise with a calculator and a blank document.

## Observe the whole task

OpenAI's observability guidance describes traces containing model calls, tool calls, handoffs, and guardrail events. Its latency guidance includes reducing requests, considering parallel work, and using non-model steps where appropriate. These are options to investigate, not guaranteed improvements for every task. [Sources: Integrations and observability](https://developers.openai.com/api/docs/guides/agents/integrations-observability), [Latency optimization](https://developers.openai.com/api/docs/guides/latency-optimization)

For this worksheet, a run begins when the request is accepted and ends when it produces a validated answer or a required stop. A correct refusal to guess is a valid outcome when the case requires it. A fast unsupported answer is a failure.

## Predeclare the comparison

The fictional task answers a studio FAQ from a fixed synthetic packet. Configuration A uses a two-step workflow. Configuration B combines the steps. The packet and expected outcomes stay unchanged; no new source, tool, or model is introduced during this paper comparison.

Write the decision rule before examining numbers: all four cases must satisfy their expected outcome, with zero boundary violations and zero unsupported material claims. Only configurations passing those gates may compete on time and cost. If neither passes, revise the design. These are local teaching criteria, not general production thresholds.

## Copy-ready run ledger

```text
Run ID / case ID / configuration version:
Fixture version / model label if actually known:
Expected outcome / observed outcome:
Start time / terminal time / elapsed milliseconds:
Human approval wait, separately recorded:
Model requests / tool calls / retries:
Input tokens / cached input tokens / output tokens:
Usage source and applicable rate source/date:
Tool or infrastructure charges:
Unsupported claims / boundary violations:
Terminal status: passed | failed | correctly_stopped | incomplete
Evidence or receipt IDs:
```

Keep token categories aligned with the provider's billing definitions. Do not double-count cached input if it is included in a total input field. When usage is unavailable, write “not measured.” Tool counts and words are workload indicators, not substitutes for billed tokens.

For a real log, collect only necessary operational metadata. Do not store credentials or sensitive raw prompts just because a tracing feature supports them. Set an access owner and retention period appropriate to the data before collecting it.

## Synthetic worked ledger

The four expected outcomes are answer, answer, answer, and stop because the source is missing.

```text
Configuration A
Case 1: 1400 ms, correct answer
Case 2: 2200 ms, correct answer
Case 3: 1800 ms, correct answer
Case 4: 1600 ms, correct required stop
Aggregate usage: 4000 input tokens, 800 output tokens

Configuration B
Case 1: 1000 ms, correct answer
Case 2: 1300 ms, correct answer
Case 3: 1200 ms, correct answer
Case 4:  900 ms, unsupported answer instead of required stop
Aggregate usage: 3000 input tokens, 600 output tokens
```

Assume a fictional input rate of USD 0.50 per million tokens and output rate of USD 2.00 per million tokens. Assume no caching, extra tool charges, or human waiting in this arithmetic fixture. These assumptions must not be carried into a real cost estimate without verification.

**A:** Sort elapsed times as 1400, 1600, 1800, 2200. Median is (1600 + 1800) / 2 = 1700 ms. Total time is 7000 ms. Token cost is 4000 × 0.50 / 1,000,000 + 800 × 2.00 / 1,000,000 = USD 0.0036. Four of four outcomes pass.

**B:** Sorted times are 900, 1000, 1200, 1300. Median is 1100 ms and total time is 4400 ms. The same hypothetical formula gives USD 0.0027. Only three of four outcomes pass.

**Decision:** Keep A as the teaching baseline and revise B's missing-source behavior. B's lower median and cost do not satisfy the predeclared quality gate. These arithmetic results describe only the invented ledger.

## Run a bounded review

1. Spend five minutes fixing the task, source packet, expected outcomes, and gate. Include one legitimate stop case so unfinished business is not automatically scored as failure.
2. Spend ten minutes completing the ledger. For a future real experiment, include failed and interrupted runs rather than silently dropping them. Label warm-up runs separately if you exclude them by a predeclared rule.
3. Spend five minutes checking calculations independently. Distinguish wall-clock time from summed component durations, especially when steps overlap.
4. Spend five minutes inspecting the slowest run and every failed case. Averages do not explain whether the delay came from tools, generation, human review, or retries.
5. Spend five minutes writing the decision record. Change only one major design variable in the next version and state which failure or cost it is intended to address.

## Copy-ready decision record

```text
Question and compared versions:
Sample size and cases included:
Predeclared quality and boundary gates:
Gate outcomes with failed case IDs:
Time summary and measurement boundary:
Actual cost or clearly labelled hypothetical calculation:
Missing measurements and confounders:
Decision: keep | revise | stop
One proposed change and reason:
Next comparison and stopping condition:
```

## Verification and limits

Verify both medians, totals, token-cost formulas, and the rejected B decision. Four cases are too few to characterize production tail latency; report the observed maximum of this small set rather than presenting it as a stable service guarantee. Do not extrapolate a monthly saving from invented or unrepresentative traffic.

Stop if versions cannot be identified, usage categories are unclear, cases differ materially, or a boundary violation occurs. Preserve incomplete rows and explain what is missing. The useful outcome is a reproducible decision with honest limits, even when the decision is “we do not have enough evidence yet.”

## Sources and scope

Sources were opened on 2026-10-13. The ledger, fictional rates, calculations, and decision rules are original educational materials. No live model timing, pricing benchmark, or optimization result is claimed.

- [OpenAI: Integrations and observability](https://developers.openai.com/api/docs/guides/agents/integrations-observability) — observable events across a run
- [OpenAI: Latency optimization](https://developers.openai.com/api/docs/guides/latency-optimization) — possible levers to evaluate against quality requirements