Define Success Before You Prompt

Why criteria come before prompts

Prompt engineering has prerequisites, and most people skip them. Before you tune a single word, you need three things: clear success criteria, an empirical way to test against them, and a first-draft prompt. Without the first two, "improving" a prompt means reading a few outputs, feeling good or bad about them, and editing on instinct. That's vibes, not engineering — you can't tell whether change #7 helped, hurt, or did nothing.

With criteria and a test set, every prompt edit becomes a measured experiment: run the eval, compare the number, keep or revert. An eval (short for evaluation) is just a fixed set of test inputs plus a way to score the outputs, so the same test can be re-run after every change.

What good criteria look like

Good success criteria are specific, measurable, achievable, and relevant to your actual use case. The test: could two people disagree about whether you hit the target? If yes, the criterion is too vague.

Quick definitions for the terms above:

• F1 — a single score (0 to 1) that balances precision (of the things you flagged, how many were right) and recall (of the things you should have flagged, how many you caught).
• ROUGE-L — a similarity score comparing a generated summary to a human-written reference, based on the longest overlapping word sequences.
• p95 latency — the response time that 95% of requests beat. It captures the slow tail, not just the average.

Notice the toxicity example: even hazy, subjective topics can be quantified. You don't need a perfect metric — you need a number that moves in the right direction when quality improves.

Evaluate along multiple dimensions

A single number rarely captures "good." Real systems get judged along several axes at once:

• Task fidelity — did it do the actual job correctly?
• Consistency — does the same input give a similar output each time?
• Relevance and coherence — is the output on-topic and logically structured?
• Tone — does it match the voice you need?
• Privacy — does it leak information it shouldn't?
• Context utilization — does it actually use the documents you gave it?
• Latency and cost — is it fast and cheap enough to ship?

You won't gate on all of these for every project, but you should consciously decide which ones matter before tuning, because prompts often trade one against another. A prompt change that improves tone might slow responses or hurt fidelity — without multi-dimensional criteria you'd never notice.

A worked example

Say you're building a prompt that summarizes customer support conversations. Before touching the prompt, define pass/fail:

A prompt version PASSES if, on the 50-conversation eval set:
1. ROUGE-L vs reference summaries >= 0.40 (fidelity)
2. LLM-judge coherence score >= 4/5 average (readability)
3. Zero summaries include customer email addresses (privacy)
4. p95 latency < 2 seconds (speed)

Now iteration is mechanical. Draft prompt scores 0.31 / 3.6 / one leak / 1.4s. You add two example summaries to the prompt: 0.42 / 4.3 / one leak / 1.8s. You add "never include contact details" with the reason why: 0.41 / 4.3 / zero leaks / 1.8s. Ship it. Every decision was a comparison of numbers, and you know exactly what each change bought you.

Design the eval set to mirror reality

Your eval should look like your real traffic, not like a demo. That means:

• Match the real task distribution — if 70% of production inputs are short and messy, 70% of eval cases should be too.
• Include edge cases deliberately: irrelevant input (a support summary prompt receiving spam), overly long input, ambiguous cases where even humans disagree, and adversarial input designed to break the rules.

A prompt that aces 20 clean, well-formed examples and has never seen a weird one will fail in production in ways your eval never predicted.

Not every failing eval is a prompting problem

If your criteria include latency or cost targets and the prompt keeps missing them, the fix may be a different model, not a better prompt. A smaller, faster model with a slightly more detailed prompt often beats a large model you're trying to prompt into being fast. Criteria keep you honest about this: because you wrote down what matters, you can see when you're optimizing the wrong lever.

The takeaway

Write the pass/fail definition before the prompt. It takes ten minutes, and it converts every future prompt edit from a guess into an experiment.