Code Grader

A code grader is a deterministic, rule-based scoring method used in evals — exact string match, regex, JSON schema validation, or running the generated code against unit tests — that grades an AI system's output without a human or another model in the loop.

It's the cheapest and most trustworthy of the three main eval scorers (alongside human review and LLM-as-judge), precisely because it trades flexibility for certainty: a code grader either finds the pattern or it doesn't, runs the tests and they pass or they don't, with no ambiguity and no run-to-run variance to argue about.

How it works

Code graders apply wherever "correct" can be checked mechanically. Common patterns: exact match against a reference answer for closed-ended questions; regex or substring checks for required elements (does the response contain a citation marker, a phone number, a specific disclaimer); JSON schema validation for structured-output compliance; and, for coding tasks specifically, executing the model's generated code against a hidden test suite and checking pass/fail. None of these require a definition of "good writing" or "helpful tone" — they require a property that's either present or absent, verifiable by running code, not by judgment.

The limitation is exactly as sharp as the strength: a code grader cannot assess anything genuinely open-ended — tone, nuance, whether an explanation is actually clear, whether a summary captured what mattered. Ask it to grade "is this a good email" and it has nothing to check. That's the gap human review and LLM-as-judge exist to fill.

Why it matters

Wherever a code grader applies, it should be the default choice: it's near-instant, free to run thousands of times in CI, perfectly reproducible, and immune to the biases and cost of an LLM judge. Mature eval suites use code graders to check the properties that admit an objective answer, and reserve the more expensive human or LLM-judge scoring for the genuinely subjective remainder — mixing scorer types rather than picking one for an entire suite.

A worked example

A team evaluates a code-generation model against a suite of 150 tasks, each with a hidden unit test suite. Before a prompt change, the code grader reports 112 of 150 tasks passing their tests — a 74.7 percent pass rate. After the change, 129 of 150 pass — 86.0 percent. Both numbers came from actually running the generated code, not from a judgment call, so an 11-point jump is a fact about behavior, not an opinion about quality — and it reruns identically on the next CI build with zero variance.

How miatz teaches it

Every coding rep in the miatz curriculum is graded by a code grader first — tests pass or they don't — before any subjective review of style or approach happens at all. Learners write their own tiny code graders early on (a regex check, a schema validator) so the mechanism scoring their work is one they understand from the inside, not a black box handing down a number.