# Compare two prompts with a fair test

Published and revised: October 13, 2026 · Version 1.0.0

A prompt comparison asks a narrow question: does a specific change improve a defined task under the conditions you tested? This guide produces a test plan, scoring sheet, and decision record. It goes deeper than saving a reusable prompt by controlling inputs, separating development from final checks, and refusing to turn a small demonstration into a universal performance claim.

## Specify the improvement before seeing outputs

Choose one task and one important failure. For example: “Does adding source IDs and unknown handling reduce unsupported statements in a short event digest?” Define success in observable terms before running either prompt.

OpenAI's evaluation guidance recommends task-specific objectives, relevant cases, explicit metrics, repeated evaluation, and human calibration of automated scoring. The small manual design below adapts those principles; it does not require an API or claim statistical validation. [OpenAI: Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices)

This exercise changes prompt text, not model weights. It is not fine-tuning, hidden-instruction modification, or a certification of an assistant's future behavior.

## 1. Freeze what stays the same

Save prompt A, prompt B, source packet, case IDs, scoring rules, and run conditions. Change one meaningful prompt feature if you want to understand its contribution. If B also changes format, length, and tools, describe the comparison as a bundle of changes.

Keep the model and settings the same where the interface exposes them. Record what the interface does not reveal rather than inventing a model identifier or configuration. Use separate clean conversations and equivalent inputs to reduce contamination from earlier answers.

## 2. Build cases with expected behavior

Start with six synthetic fixtures: ordinary facts, missing venue, contradictory approved capacity, changed source revision, embedded instruction to send a message, and an unsupported request for a attendance forecast. Write an expected outcome for each before generation.

Use four for development and keep two unseen during prompt revision. This small held-out set is a useful discipline, not strong evidence of generalization. Keep cases separate from example text embedded in the prompt; a copied example can make the result look better than it is.

## 3. Separate hard gates from graded quality

Hard gates should cover unacceptable outcomes: fabricated essential facts, missed material conflicts, or actions beyond permission. A polished answer cannot compensate for a failed gate.

Then score limited dimensions, such as evidence coverage and reader usability, on a short anchored scale. Define what 0, 1, and 2 mean. Record correction time separately if that matters, using the same reviewer task. Do not mix seconds, ratings, and failure counts into a mysterious total.

## 4. Reduce avoidable judging bias

Label outputs X and Y without revealing the candidate. Alternate their display order. Have a human compare them against the source packet and rubric. If a model helps grade, verify its judgments against human-reviewed examples; a model's confidence is not independent evidence.

Plan repetition because outputs can vary. A bounded first comparison could allow two outputs per candidate per case: 24 outputs total. Keep the four development cases and two final cases distinguishable in the report. If you change a prompt, assign a new version and identify which checks need rerunning.

## Copy-ready comparison directive

```text
Prepare, but do not claim to have executed, a comparison of prompts A and B.
Task: [narrow job]. Proposed change: [single feature or named bundle].
Sources: [fixed packet/version]. Run budget: [outputs/time/spend].
Conditions to record: [visible model/settings, tools, date, source versions].

Create six synthetic cases, four development and two held out for final review.
For each, state expected behavior and unacceptable failures before outputs exist.
Define hard gates plus two anchored quality scales, each scored 0/1/2.
Plan blinded labels, alternating order, and repeated runs within the budget.

Result sheet: case ID, prompt version, repeat, output reference, each gate,
each quality score, reviewer rationale, correction time, and unknown conditions.
Keep development and held-out results separate. Report denominators.
Do not claim significance, broad superiority, or deployment readiness from
this small test. Do not publish, call paid tools, or change live workflows.
Stop for missing sources, unapproved cost, or consequential permission failures.
```

## Synthetic worked example: the missing venue

Fixture E2 says: “Two sessions; maximum 30 people each; 40 minutes per session. Venue undecided.”

Prompt A: “Write a friendly event digest in three bullets.”

Prompt B: “Write a friendly event digest in three bullets. Preserve source facts, attach the fixture ID to each factual bullet, and label missing logistics as undecided.”

Consider these authored sample outputs, not actual model-run evidence:

- X: “Join two 40-minute sessions at the town hall, with 30 places each”
- Y: “Two 40-minute sessions are planned, with 30 places each [E2]. Venue: undecided [E2]”

X fails the essential-fact gate because town hall is invented. Y preserves the missing venue and supplied quantities. Under an evidence scale where 0 means missing or incorrect support, 1 means partial support, and 2 means complete support for factual claims, these samples receive 0 and 2 respectively.

That result illustrates the rubric only. It does not show that prompt B will always produce Y, that A always invents venues, or that the extra instruction is the sole cause of a difference.

## Checks and stop criteria

Before choosing a candidate, verify complete records, identical inputs, declared variations, stable scoring, and separate final-case results. Report “3 of 4 outputs passed” rather than “75% reliable” without context. Investigate any hard-gate failure even when average quality improves.

Stop when the budget is exhausted, comparisons are contaminated, essential conditions changed, or a result requires new permission. Keep the simplest candidate that meets the declared gates, or retain the baseline if the evidence is inconclusive. Record what further test would change that decision.