Temperature Sampling

Temperature is a number, usually between 0 and 2, that rescales a language model's predicted probabilities before it samples the next token — low temperature sharpens the distribution toward its most likely choice, high temperature flattens it toward more uniform randomness.

At every step of generation, a model does not output one token — it outputs a probability distribution over its entire vocabulary, one score per possible next token. Sampling is the step that turns that distribution into an actual choice. Temperature controls how sharply that choice favors the top candidates versus how much of a chance the long tail of less-likely tokens gets.

How it works

Before the model's raw scores (logits) are converted into probabilities via softmax, each one is divided by the temperature, T. Dividing by a small T (below 1) exaggerates the gaps between logits, so the already-strongest candidate gets pushed even further ahead; dividing by a large T (above 1) shrinks the gaps, pulling weaker candidates closer to competitive. At T equal to 0, sampling becomes deterministic — the model always takes the single highest-scoring token, known as greedy decoding.

Take four candidate next tokens with raw scores 2.0, 1.0, 0.5, and 0.1. At T = 1.0 (no rescaling), softmax gives them roughly 57%, 21%, 13%, and 9%. Drop to T = 0.5 and the same four scores become roughly 83%, 11%, 4%, and 2% — the leader now dominates almost completely. Raise it to T = 1.5 and they flatten to roughly 46%, 24%, 17%, and 13% — genuinely competitive odds for every option. Same underlying model, same four candidates, three entirely different behaviors from one dial.

Why it matters

Temperature is the main control for the accuracy-versus-creativity trade-off in generation. Low temperature (0 to 0.3) suits tasks with one correct answer — code generation, data extraction, classification, math — where you want the model's best guess every time and randomness only adds errors. Higher temperature (0.7 to 1.0+) suits brainstorming, creative writing, and varied phrasing, where multiple different outputs are all acceptable and some diversity is the point. Set it wrong in either direction and you get a visible failure: a support bot at temperature 1.2 that gives a different, sometimes wrong answer to the identical question every time, or a creative-writing assistant at temperature 0 that produces the same flat, most-probable sentence on every retry.

A worked example

The four-candidate example above is the full worked case: raw scores 2.0, 1.0, 0.5, and 0.1 become probabilities of about 57%, 21%, 13%, and 9% at T=1, about 83%, 11%, 4%, and 2% at T=0.5, and about 46%, 24%, 17%, and 13% at T=1.5. Notice temperature never changes the ranking — the top candidate stays on top at every setting — it only changes how much of a chance the alternatives get to be sampled instead.

How miatz teaches it

The AI Labs playground includes a temperature lab: learners fix a prompt, sweep temperature from 0 to 1.5, and generate a batch of outputs at each setting to watch determinism collapse into diversity firsthand, building the intuition to pick the right value by task rather than by habit.