Eval Gate

An eval gate is an automated check, wired into CI/CD, that blocks a deployment or merge unless an eval suite's scores meet or exceed a defined threshold — the same discipline a test suite enforces for code, applied to the probabilistic output of an LLM-based system.

Traditional CI has always gated on tests passing; an eval gate extends that same discipline to the parts of an AI system a unit test can't check — groundedness, retrieval quality, judged helpfulness — so a prompt edit, model swap, or retrieval tweak has to clear a measured bar before it reaches production, the same as any other code change.

How it works

A team defines thresholds per metric based on the current baseline and acceptable risk — for example, groundedness must stay at or above 90 percent and citation rate at or above 85 percent. Every pull request that touches a prompt, retrieval configuration, or model version reruns the full eval suite against the golden dataset, and the gate compares the new scores against the thresholds. Meet every threshold and the check passes; miss even one and the merge is blocked, regardless of how much better the change looks on the examples someone tried by hand.

The thresholds themselves need periodic revisiting — set them too loosely and regressions slip through; set them as a ratchet that only ever tightens and legitimate trade-offs (a slight groundedness dip in exchange for a large latency win) become impossible to ship deliberately. Most mature setups also require a human override path with a name and a reason attached, for exactly those deliberate trade-offs.

Why it matters

Without an eval gate, a prompt or model change is only as safe as whoever reviewed it remembered to test — and prompt changes are code that doesn't look like code, so they get less scrutiny than a function signature change would, right up until they break something in production. An eval gate makes regressions a build failure instead of a support ticket, catching the class of bug that is otherwise invisible until users hit it.

A worked example

A team's eval gate requires groundedness ≥ 90 percent and citation rate ≥ 85 percent. A new prompt version scores 92 percent groundedness — a real improvement over the 88 percent baseline — but citation rate drops to 81 percent, below its threshold. The gate fails the build and blocks the merge, even though the headline metric improved, because the prompt change that boosted groundedness also made the model less consistent about citing where its claims came from. The engineer traces it to an instruction that got dropped in editing, restores it, reruns the suite, and ships once both thresholds clear.

How miatz teaches it

Capstone projects wire a real eval gate into their own CI pipeline as a graded requirement, not an optional extra — learners experience, at least once, a build failing on an eval regression they didn't notice by eye, which is the entire argument for gates in one lived incident.