What do temperature and top-p do in GPT-3?

Every time GPT-3 writes a token, a word or piece of a word, it first produces a score for every token it knows: how likely each one is to come next. The model itself stops there. Something still has to choose one token from that list, and the rules for that choice are the settings you see beside the prompt box in OpenAI's API, chief among them temperature and top-p.

These two numbers don't make the model smarter or change what it knows. They change how boldly it picks. The same prompt can give a crisp, repeatable answer or a different, surprising story every time, depending on how you set them.

A dice analogy

Picture the model's choice as rolling a loaded die. Each face is a possible next word, and the faces are different sizes: likely words get big faces, unlikely words get tiny ones.

• Temperature changes how loaded the die is. At a low temperature, the big faces grow and the small ones shrink, so the favorite comes up almost every time. At zero, there is no roll at all: the model always takes the top word. At a higher temperature, the faces even out and long shots come up more often.
• Top-p removes small faces from the die altogether. With top-p set to 0.9, the model keeps the most likely words until together they cover 90% of the probability, throws the rest away, and rolls among what remains.

The Hugging Face team's guide to text generation describes the same two ideas in technical terms. Lowering the temperature sharpens the probability distribution. Top-p, also called nucleus sampling, picks from the smallest group of words whose combined probability passes p, so the group is large when many words could fit and small when the next word is obvious.

A worked example in numbers

Take the prompt "The weather today is" and suppose the model scores five candidate words as below. These starting scores are invented for illustration; the other columns are calculated from them with the standard temperature formula.

At temperature 0, the model writes "sunny" every single time. At 0.7, "sunny" is still the favorite but "cloudy" and "rainy" appear regularly, and "purple" almost never. At 1.0, the model uses its scores as they are, so the odd word "purple" turns up about three times in a hundred.

Now add top-p. With top-p at 0.8, the model keeps "sunny", "cloudy" and "rainy", which together reach 90% at temperature 1.0, and drops "perfect" and "purple" entirely. The remaining three are rescaled to about 56%, 28% and 17%. The strange option is gone, but there is still variety among sensible ones.

One "purple" in a sentence looks harmless. But the choice repeats for every token, and each unusual pick becomes part of the text the model builds on. A few long shots in a row can carry a paragraph somewhere odd.

What GPT-3's own researchers used

The GPT-3 paper gives a concrete reference point. For its sample poems and for some of its bias measurements, the authors generated text at temperature 1 with nucleus sampling at p = 0.9. That combination keeps the model's natural variety while trimming the least likely 10% of options.

Users in the API beta have settled into a similar split. Developer Max Woolf, writing in July after testing GPT-3, describes two broad uses: creative text at a high temperature, and functional tasks, such as producing web page mock-ups, at a temperature of 0.0. He also found that offensive output became more noticeable at higher temperatures, a reminder that wider sampling lets more of the web's worst writing through.

A practical tip: change one setting at a time. Temperature and top-p both widen or narrow the choice, so moving both at once makes it hard to tell which one caused a change.

Which settings should you use for facts and for stories?

For facts, classification and anything with one right answer, use a low temperature, 0 or close to it. You get the model's single most likely answer, and the same prompt gives the same reply, which makes testing much easier. The guide from Hugging Face warns that pure top-word picking can fall into repetition in long text, so keep outputs short for these tasks. Keep in mind that a low temperature makes answers consistent, not correct: if the model's top guess is wrong, it will be wrong every time.

For stories, brainstorming, names and dialogue, try a temperature around 0.7 to 0.9, or a temperature of 1 with top-p around 0.9, as the GPT-3 authors did for poetry. You will get variety from run to run, so generate several versions and keep the best. If the writing starts to wander or contradict itself, lower the temperature a step before rewriting the prompt.

Between those two poles there is no single right number. A product description may want a little life, such as 0.5, while a summary of a contract wants none. Start from the use, test on a few real prompts, and write down the settings that worked so you can repeat them.