Peek at Your Own Risk

Run an A/B test where there is no real difference — two identical arms — and watch what happens when you keep checking the results early. Each peek is another chance for random noise to cross the 5% significance line. Drag the number of peeks and see the true false-positive rate balloon far past the 5% you thought you were running. Simulated live in your browser (0 uploads, works offline).

How to use it

Set how many times you peek at the results before the test is done.
Compare the two bars — the 5% you assumed versus the actual false-positive rate across thousands of simulated no-effect tests.
Read the X-ray: exactly how much stopping early on the first green result inflates your error rate, and why.

What this clears up (the fundamentals)

Optional stopping is a multiple-comparison problem — a test fixed at 5% assumes one look at a pre-set sample size. Every extra peek is another independent shot at a false positive, so the errors compound.
Noise wanders — with no real effect, the observed difference drifts up and down. Given enough looks, it will cross the threshold by chance; peeking lets you stop exactly when it does.
The inflation is large and fast — even a handful of peeks can double or triple the 5% error rate. "We checked and it was significant" is not the same as "it's significant."
The fixes are known — fix the sample size in advance, or use a sequential/alpha-spending method designed for continuous monitoring. Watching a dashboard and stopping when it's green is neither.

Where it's used

An experimentation-literacy drill — the fastest way to feel why "we saw significance on day 3 and shipped" is a trap, and why analytics tools warn against early stopping. It's a miatz build-lab concept playable; learn to build it yourself.

FAQ

What is an A/A test?

An A/B test where both arms are identical — there is genuinely no difference. Any "significant" result it produces is by definition a false positive, which makes it the perfect way to measure how often your process cries wolf.

Why does peeking inflate the false-positive rate?

Because each look is another opportunity for random fluctuation to exceed the significance threshold. The 5% guarantee applies to a single test at a fixed sample size; checking repeatedly and stopping at the first significant result is testing many times and keeping the luckiest look.

How is this simulated?

Thousands of A/A experiments run entirely in your browser: each accumulates samples for two identical arms and runs a z-test at every checkpoint, stopping if any crosses |z| > 1.96 (the 5% two-sided line). The false-positive rate is the fraction that stopped early.

Is anything uploaded?

No. Every simulated trial runs on your device — nothing is transmitted or stored. Turn off your Wi-Fi and it still works.

Limits

A teaching simulation with equal-variance normal data and evenly spaced looks — real tests have skewed metrics, unequal arms, and business-driven stopping. The direction and rough magnitude hold regardless: more peeks, more false positives. Use a proper sequential-testing tool for real decisions.

Part of the miatz playgrounds. Explore the rest from the playground catalog.

Bookmark this page (Ctrl+D, or ⌘D on Mac) — it works offline the next time you need it.

Peek at your own risk

How to use it

Set how many times you peek at the results before the test is done.

Compare the two bars — the 5% you assumed versus the actual false-positive rate across thousands of simulated no-effect tests.

Read the X-ray: exactly how much stopping early on the first green result inflates your error rate, and why.

What this clears up (the fundamentals)

Optional stopping is a multiple-comparison problem — a test fixed at 5% assumes one look at a pre-set sample size. Every extra peek is another independent shot at a false positive, so the errors compound.

Noise wanders — with no real effect, the observed difference drifts up and down. Given enough looks, it will cross the threshold by chance; peeking lets you stop exactly when it does.

The inflation is large and fast — even a handful of peeks can double or triple the 5% error rate. "We checked and it was significant" is not the same as "it's significant."

The fixes are known — fix the sample size in advance, or use a sequential/alpha-spending method designed for continuous monitoring. Watching a dashboard and stopping when it's green is neither.