Golden Dataset

A golden dataset (or gold set) is a curated collection of representative inputs paired with human-verified expected outputs or labels, used as the ground truth an eval suite scores a system's actual outputs against.

Every metric this glossary defines elsewhere — recall@k, precision@k, MRR, groundedness rate — is meaningless without one: they all require knowing, for each example, what the correct answer or correct set of relevant documents actually is. That "correct" has to come from somewhere trustworthy, and a golden dataset is where it lives.

How it works

Building one well means starting from real production traffic and real failure reports rather than inventing hypothetical questions, because hand-written examples systematically miss the messy, ambiguous inputs actual users send. Each example gets a reference answer, a relevance label, or a rubric score, ideally produced or reviewed by a subject-matter expert rather than whoever is fastest to type something plausible. Multiple annotators labeling a subset and checking their agreement rate is standard practice — low agreement on a slice of examples usually means the labeling instructions were ambiguous, not that the annotators were careless.

Size matters more than it first appears. A golden set's job is to estimate a system's true pass rate with enough precision to trust a before/after comparison, and that precision follows directly from sample size: for a system passing around 80 percent of examples, a 95 percent confidence interval is roughly ±14.3 percentage points wide at 30 examples, but narrows to roughly ±5.5 points at 200 examples. A 30-example golden set can make a real 5-point regression indistinguishable from noise.

Why it matters

A golden dataset is what turns "the new prompt seems better" into a claim someone can actually check. Without one, every model upgrade, prompt edit, or retrieval change is judged by whichever handful of examples someone happened to try — which reliably favors changes that fix the examples in front of you while silently breaking ones you didn't retest. It's also the resource that gets stale fastest: a golden set frozen at launch stops reflecting how the product and its users have actually evolved, so mature teams keep adding real failure cases to it over time.

A worked example

A team builds a 200-example golden set for a support RAG bot, each example carrying a reference answer and the specific source documents that should be retrieved. A prompt change scores 82 percent groundedness against it, versus a 79 percent baseline — a 3-point gain. With this sample size, the 95 percent confidence interval on each score is roughly ±5.5 points, wide enough that a 3-point difference alone shouldn't be trusted; the team needs either a larger golden set or a paired-comparison statistical test before shipping the change on this number alone.

How miatz teaches it

Learners build their own 30-example golden set for a capstone RAG project, then watch how unstable its pass-rate estimate is against a held-out 200-example set assembled from the same source — a direct, uncomfortable lesson in why eval sample size is not a formality.