Temperature: the knob that decides whether AI makes things up

A client once showed me two answers his assistant gave to the same question. One was perfect; the other offered the customer a discount that didn't exist. Same question, same model, same day. "How is that possible?", he asked. The answer has a name: temperature.

How a model picks a word

A language model doesn't "know" the answer and write it down. It looks at all the previous text and computes, for every possible word, how likely it is to come next. Then it picks one. Not always the most likely: it draws one, with better odds for the higher-scoring words.

Temperature controls how much those odds get sharpened or flattened before the draw. At low temperature, the favorite almost always wins. At high temperature, the odd options start to have a real chance.

Turn the knob and generate a few answers:

Which temperature to use

There's no right value: it depends on what you ask the model for.

• Between 0 and 0.3 to answer with facts: prices, hours, policies, pulling data out of a document. There you want the same answer every time.
• Between 0.5 and 0.8 to converse: a customer-facing assistant sounds more natural if it doesn't repeat itself, without drifting.
• Above 1 to generate ideas: names, titles, ad variations. There, the odd answer is exactly what you're after.

Temperature doesn't fix everything

Lowering the temperature reduces variety, not mistakes. If the model lacks the right data, at temperature 0 it will say the same wrong thing every time, with total confidence. What stops it from making things up is giving it the true information at answer time and telling it explicitly what to do when it doesn't know.

My client's assistant had its temperature at 1.2 "so it sounds more human". We lowered it to 0.4, gave it the real list of promotions, and the made-up discounts disappeared.