Calibration

Calibration is the match between how confident you say you are and how often you're actually right. A perfectly calibrated forecaster who says "90% sure" is correct on almost exactly 90% of those claims — no more, no less.

Most people are not calibrated, and almost nobody knows it, because everyday life rarely forces the comparison. You state confidence constantly — "this will work," "I'm sure that's the right file," "the deploy is safe" — but you almost never go back and score those statements against outcomes. Without the reconciliation step, confidence is just a feeling.

How it's measured

Calibration is measured by eliciting a probability with each prediction, recording the outcome, and comparing the two across many predictions. The standard visualization is a calibration curve: stated confidence on one axis, observed accuracy on the other. A calibrated forecaster's points sit on the diagonal; an overconfident one's points sag below it (said 90%, was right 60%). The standard single-number score is the Brier score — the mean squared gap between stated probability and outcome.

Why it matters for working with AI

Every interaction with an AI system involves an implicit confidence call: you decide whether to accept an answer, merge generated code, or let an agent act. Research on automation bias shows fluent output inflates confidence — polished prose feels correct — which is exactly the failure calibration training targets. And unlike most cognitive skills, calibration has strong evidence of trainability: Philip Tetlock's forecasting-tournament research found that people who repeatedly make probability predictions and confront the outcomes durably improve their judgment, and that the improvement transfers across domains.

The training loop is simple: before a consequential decision, state one falsifiable prediction with a probability; afterwards, score it honestly; over weeks, watch your curve. Surprise — a high-confidence miss — is the most valuable data point, because it marks exactly where your model of the world diverges from the world.

Where you meet it in miatz

The free Calibration Lab gives you a first curve from twenty general-knowledge questions in five minutes. Miatz Tuukul then builds the real one from your actual work: a morning prediction about the day's biggest AI delegation, an evening reconciliation, and a curve computed from months of genuine outcomes rather than trivia.