Evals Runner
Run one golden set of AI outputs through two graders at once — a deterministic code grader (a cheap keyword rule) and an AI judge (a holistic read of correctness) — and slide a single strictness threshold to see exactly where they disagree. The moat is that disagreement view: instead of trusting either grader alone, you see the precise cases where the code rule and the model’s intuition part ways, which is the exact ambiguity a rubric has to resolve rather than paper over. Runs entirely in your browser (0 uploads, works offline).
How to use it
- Drag the judge strictness slider, or load a preset — Lenient, Balanced, or Strict judge.
- Read the per-case grid: each row shows the model output, the Code verdict, the Judge verdict, and a ✓/✗ for whether the two agreed.
- Watch the agreement rate and disagreement count update live as you move the threshold.
- Open the X-ray panel to see the disagreement heat-map — each split case with a plain-English reason the graders diverged.
What this clears up (the fundamentals)
- A code grader and an LLM judge measure different things — the code grader here checks a literal keyword; the judge reads meaning. Agreement is not guaranteed, and the gap is the whole lesson.
- Disagreements are a feature, not noise — every case where the two graders split is exactly where your rubric is under-specified. Papering over it with one grader hides the ambiguity instead of resolving it.
- Graders get hacked in both directions — a reply can match the keyword rule while being wrong (grader-hacked), or be correct while dodging the keyword (a miss). Both show up as disagreements.
- Strictness is a dial, not a fact — moving the judge’s pass threshold changes who passes; a threshold that makes both graders agree can simply be hiding the hard cases.
Where it's used
The fastest way to feel why “just use an LLM judge” isn’t enough — the disagreement heat-map explains it to a skeptical exec in ten seconds. It’s a miatz build-lab concept playable: play the grader-vs-judge comparison here, then learn to build the golden set, the code grader, and the judge rubric yourself.
FAQ
What's the difference between a code grader and an LLM-as-judge?
A code grader is deterministic — it applies an exact rule (here, “does the reply contain the keyword?”) and always returns the same verdict. An LLM-as-judge scores holistic quality and can catch meaning a keyword rule misses — but it’s fuzzier and can be fooled. Real eval suites use both and reconcile them with a rubric.
Why do the graders disagree?
Because they measure different things. A reply can satisfy the literal rule while being wrong (a grader-hack), or be correct while phrasing around the keyword (a miss). Each such case is a row where the code verdict and the judge verdict differ.
What does the strictness slider actually change?
It sets the judge’s pass threshold: a case passes the judge when its quality read is at or above the threshold. Slide it to 0 and the judge passes everything; slide it to 1 and it fails everything. The interesting behavior is in the middle, where disagreements with the code grader appear.
How is the agreement rate calculated?
For each case the engine compares the code verdict to the judge verdict at the current strictness, counts how many agree, and divides by the total number of cases (rounded to three decimals). It’s a deterministic calculation — no model call, no randomness.
Is anything uploaded?
No. The golden set and both graders run entirely in your browser — nothing is transmitted, stored, or logged. Turn off your Wi-Fi and it still works.
Limits
This is a teaching harness with a small, fixed golden set and precomputed, illustrative judge scores baked into the page — it is not calling a live model. Real evals route outputs through an actual judge model whose scores vary, use larger golden sets, and cost tokens per run. The mechanism it demonstrates (compare a code grader to a judge and treat disagreements as your rubric to-do list) is exactly the real practice.
Related
Part of the miatz playgrounds. Explore the rest from the playground catalog.
Bookmark this page (Ctrl+D, or ⌘D on Mac) — it works offline the next time you need it.