How to Build a Golden Eval Set for an AI Feature

Before you tune a prompt or swap a model, build a golden eval set: a fixed collection of real inputs with known-good outputs that you score against every time you change something. Without it, you are flying blind. You tweak the prompt, the demo looks better, you ship, and you have no idea whether you improved the feature or just got lucky on the three examples you happened to try. The golden set is the difference between engineering and vibes. It is the cheapest reliability investment in AI-native development, and almost everyone skips it until a regression burns them.

The thesis: you cannot improve what you do not measure, and for an AI feature, measurement means a golden eval set you built by hand and trust.

What goes in a golden eval set

Real inputs, not invented ones. Pull from actual usage or actual documents. Synthetic examples miss the weird formatting, the typos, the edge cases that break things in production. Fifty real cases beat five hundred clean fake ones.

Cover the distribution, then oversample the hard cases. Include the common happy path, but deliberately load in the ambiguous inputs, the ones near a decision boundary, the ones a human would pause on. Those are where models fail and where a change actually moves the score. This is the same instinct as writing characterization tests before you refactor: lock down the tricky behavior first.

Label the expected output, or at least the acceptance criteria. For extraction, that is the exact right value. For a summary or a draft, an exact string is wrong, so write the criteria instead: must mention X, must not invent Y, must stay under Z words. Fuzzy tasks need rubric-style labels, and that is fine.

Keep it small enough to actually review and big enough to be stable. Somewhere between 50 and a few hundred cases for most features. Small enough that you can eyeball a full run, big enough that one flaky case does not swing the number.

How to score against it

Decide your metric per case type. Exact match for structured extraction. A rubric or LLM-as-judge score for open-ended output, used carefully. A human spot-check on a sample every time, because automated scores drift from what you actually care about.

Run the whole set on every meaningful change: new prompt, new model version, new retrieval strategy. Record the score. A change that raises the demo but lowers the golden-set score is a regression wearing a costume. This is how you catch that every model upgrade can quietly erase a capability edge or introduce a new failure, before your users do.

Freeze the set. The whole point is that it does not move, so score changes come from your system, not from a shifting benchmark. When you add cases, version the set so old and new scores stay comparable.

When a golden set is not enough

The caveat, so you do not over-trust it. A golden eval set measures the cases you thought to include. Production will surface inputs you never imagined. So the golden set is your offline gate, not your ground truth, and it pairs with production monitoring rather than replacing it. New failure modes found in the wild feed back into the set, which grows and hardens over time.

It also does not tell you if the feature is worth building. It tells you if this version is better than the last version. Judgment about the product still sits with you.

Why this is non-negotiable in AI-native work

Deterministic code has tests. AI-native features have eval sets, and they play the same role: a regression alarm that lets you change things fast without breaking what worked. Skipping it is how teams end up afraid to touch a prompt because nobody knows what will break. A reliable AI feature is one you can measure, and the golden set is the ruler.

We build one per model-backed feature across the Girard AI stack before we let it near a customer, and we grow it every time production shows us something new. Build the set first. Tune second. Ship third. Any other order is guessing.