The goal of this article is to explain what temperature is in LLMs and how it changes the behaviour of the output of the LLMs.

I am focusing on this topic because temperature has become a critical lever in modern prompt engineering. A less-known parameter that users now frequently adjust to balance precision against creativity. To provide a clear, technical foundation, this article focuses exclusively on the role of temperature during inference, where it functions as an active control knob rather than a fixed model weight.

At its core, a decoder-only model is a next-token predictor. Given a sequence of words (or sub-words), it calculates which token should come next. The process works in three quick steps:

  • Scoring: The model’s final layer produces a raw score for every possible word in its vocabulary—a list that can easily reach tens of thousands. These raw scores are called logits.

  • Conversion: Those logits are converted into a probability distribution, where each word is assigned a percentage likelihood.

  • Sampling: Finally, the model samples from these probabilities to pick the actual word it will output.

This article explores the specific mechanism that performs this conversion and the temperature parameter that allows you to control how the model makes those choices. Before we get into temperature, however, we need to look at the Softmax function and how it shapes the model’s decision-making process

The Softmax Function and Its Properties

Suppose $n \in \mathbb{N}$ represents our vocabulary size, and $v \in \mathbb{R}^n$ is a vector of raw scores (called logits). We can represent $v$ as an $n$-tuple:

\[v = (v_1, v_2, \ldots, v_n)\]

with each component $v_i \in \mathbb{R}$.

The Softmax function $S:\mathbb{R}^n \to (0,1)^n$ normalizes these scores into values between 0 and 1:

\[S(v_i) = \frac{e^{v_i}}{\sum_{j=1}^n e^{v_j}}\]

One immediate observation is that the Softmax function outputs a valid probability distribution, meaning the elements are all positive and sum up to 1. It transforms any arbitrary vector of real numbers into a probability vector.

Transformer decoder layer
Model Architecture: The final layers of a decoder LLM, highlighting the Linear → Softmax pipeline that prepares token outputs.
Geometric softmax mapping
Geometric Intuition: A 3D visualization showing raw logits (blue) being compressed into a bounded probability simplex (red).

Now that we have established how the Softmax function turns arbitrary logits into a valid probability distribution, we can look at how the model actually uses this probability map to generate text during inference

The Token Generation Process in LLMs

In a decoder-only LLM, the main objective is to predict the next token (word or sub-word) in a sequence.

Right before the final output stage, the model’s last linear layer produces a vector of unnormalized scores (logits) of the size of the vocabulary, where each element corresponds to a specific word in our vocabulary. If we wanted the model to pick only the single absolute best token, we would theoretically want an output vector that contains a 1 at the index of that best word and a 0 everywhere else.

Mathematically, this hard selection is represented by the Argmax function:

\[\text{Argmax}: \mathbb{R}^n \to \{0,1\}^n\] \[\text{Argmax}_i(v) = \begin{cases} 1 & \text{if } v_i = \max\{v_1, \dots, v_n\} \\ 0 & \text{otherwise} \end{cases}\]

However, during training, the model needs to learn via gradient descent. The Argmax function is a step function; its derivative is zero almost everywhere, and it is completely discontinuous at the boundaries. Because it is not differentiable, we cannot backpropagate errors through it to update the model’s weights.

This is exactly why we use Softmax. It acts as a “soft”, continuous, and fully differentiable approximation of Argmax, turning raw scores into a smooth probability map that allows the network to train while still identifying the most likely next tokens.

Note: Despite its name, Softmax is not actually a soft approximation of the maximum function (in fact the soft approximation of the maximum function is LogSumExp: $\log \sum e^{x_i}$). Instead, it is a soft approximation of the Argmax function, which is why many researchers prefer the more accurate term Softargmax.

Soft (smooth) approximation

What does it mean when we say Softmax is a soft approximation of Argmax?

In simple terms, we are about to prove two extremes: if we drop the temperature to zero, the model becomes completely deterministic (Argmax). If we crank it to infinity, the model becomes completely random (Uniform Distribution). In the following we first mathematically define approximation and prove exactly how these two extremes happen.

Here are two ways to say that a sequence of functions $f_1, f_2, \ldots$ (each mapping $\mathbb{R}^n \to [0,1]^n$) gets closer to a target function $f$.

  • Pointwise convergence: Fix one input $x$. As $k$ grows, the output $f_k(x)$ approaches $f(x)$.

  • Uniform convergence: The approximation is close everywhere at once. For large enough $k$, $f_k(x) \approx f(x)$ for every input $x \in \mathbb{R}^n$ — not just for one $x$ at a time.

Note: Uniform convergence is the stronger notion: uniform $\Rightarrow$ pointwise, but not the other way around. We do not dig deep into the concept of convergence since the main objective of this article is about the role of softmax in LLMs. However the curious reader can refer to this very illustrative video: here.

Now suppose for each $T \in \mathbb{R}$ we define,

\[\begin{gather} S_T:\mathbb{R}^n \to (0,1)^n \\ S_T(v_i) = \frac{e^{v_i/T}}{\sum_{j=1}^n e^{v_j/T}} \end{gather}\]

$S_T$ in the above equation is called the softmax function with temperature $T$.

Note: It is worth being precise about when temperature plays a role. During training, the standard Softmax (implicitly with $T = 1$) is used to maintain differentiability and enable gradient-based learning. During inference, however, temperature becomes a tunable knob that controls the shape of the output distribution after the model’s weights are frozen. Unless stated otherwise, every reference to temperature in this article refers to the inference-time setting.

$\boldsymbol{S_T \stackrel{T \to 0}{=} {\text{Argmax}}}$

Let $M = \max_{j} v_j$ be the maximum value in the vector $v$. We want to prove that as $T \to 0^+$, the softmax distribution pointwise converges to the argmax distribution:

\[\lim_{T \to 0^+} S_T (v)_i = \text{Argmax}(v)_i\]

Let’s evaluate the limit of the $i$-th component, $S_T(v_i)$:

\[\lim_{T \to 0^+} S_T(v_i) = \lim_{T \to 0^+} \frac{e^{v_i/T}}{\sum_{j=1}^n e^{v_j/T}}\]

We multiply both the numerator and the denominator by $e^{-M/T}$:

\[S_T(v_i) = \frac{e^{v_i/T} \cdot e^{-M/T}}{\left(\sum_{j=1}^n e^{v_j/T}\right) \cdot e^{-M/T}}\]

Using the exponent rule $e^a \cdot e^b = e^{a+b}$, we can rewrite this as:

\[S_T(v_i) = \frac{e^{(v_i - M)/T}}{\sum_{j=1}^n e^{(v_j - M)/T}}\]

In the denominator there is one index, like $k$, such that $v_k = M$. So, $(v_k - M) = 0$. Therefore, $e^{(v_k - M)/T} = e^{0/T} = e^0 = 1$. For other indices, we know $v_j < M$, so $(v_j - M)$ is a strictly negative constant. Let $c_j = v_j - M < 0$. As $T \to 0^+$, the exponent $\frac{c_j}{T} \to -\infty$. Consequently, $e^{(v_j - M)/T} \to 0$.

Thus, the limit of the denominator as $T \to 0^+$ is:

\[\lim_{T \to 0^+} \sum_{j=1}^n e^{(v_j - M)/T} = 1+0 = 1\]

Now, we evaluate the limit of the entire fraction by looking at the numerator for the two possible cases for $v_i$:

Case 1: $v_i$ is not the maximum ($v_i < M$)

If $v_i < M$, then $(v_i - M) < 0$.

\[\lim_{T \to 0^+} e^{(v_i - M)/T} = 0 \Longrightarrow \lim_{T \to 0^+} S_T(v_i)= \frac{0}{1} = 0\]

Case 2: $v_i$ is the maximum ($v_i = M$)

If $v_i = M$, then $(v_i - M) = 0$.

\[\lim_{T \to 0^+} e^{(v_i - M)/T} = e^0 = 1 \Longrightarrow \lim_{T \to 0^+} S_T(v_i)= \frac{1}{1}= 1\]

Combining both cases, we get: \(\lim_{T \to 0^+} S_T(v_i)= \begin{cases} 1 & \text{if } v_i = \max(v) \\ 0 & \text{otherwise} \end{cases}\)

The above argument shows that the softmax function converges pointwise to the argmax function. This is why setting temperature $= 0$ in an API call gives you the exact same response every time. The distribution collapses into a single point, forcing the model to always greedily pick the most likely token.

Note: However, this convergence is not uniform. In fact, since all of the $S_T$ functions are continuous, if they uniformly converged to a limit function $S$, then $S$ must be continuous too. However, the Argmax function is not continuous. In practical terms, this means that the “sharpening” effect of lowering the temperature is input-dependent. For a logit vector where one score strongly dominates the others, even a moderate temperature reduction will push the distribution close to deterministic. But for a logit vector where scores are nearly tied, you may need to lower the temperature much further to achieve the same level of concentration. There is no single threshold temperature that produces the same behavior across all inputs. Simply put: context always matters.

$\boldsymbol{S_T \stackrel{T \to \infty}{=} \frac{1}{n}}$

Let us prove the claim: by increasing the temperature, the functions $S_T$ converge to the uniform distribution $S(v)_i=\frac{1}{n}$.

Here is the mathematical proof.

The Uniform Distribution: A discrete uniform distribution over $n$ items means each item has an equal probability of occurring. The probability for each item $i$ is exactly the “mean” or equal share of the total probability mass ($1$):

\[U(\mathbf{x})_i = \frac{1}{n}\]

For each $1 \leq j \leq n$,

\[\lim_{T \to \infty} e^{v_j/T} =e^{\lim_{T \to \infty} v_j/T} = e^0 = 1\]

since $v_j$ is a constant real number.

Now, we apply this to both the numerator and the denominator of our softmax function.

For the numerator:

\[\lim_{T \to \infty} e^{v_i/T} = 1\]

For the denominator: Because the limit of a finite sum is the sum of the limits, we get:

\[\lim_{T \to \infty} \sum_{j=1}^n e^{v_j/T} = \sum_{j=1}^n \left( \lim_{T \to \infty} e^{v_j/T} \right) = \sum_{j=1}^n 1\]

Summing the number $1$ exactly $n$ times simply gives us $n$:

\[\sum_{j=1}^n 1 = n\]

Putting the numerator and denominator back together, the limit of the entire fraction is:

\[\lim_{T \to \infty} S_T(v)_i = \frac{1}{n}\]

This proves that as the temperature approaches infinity, the softmax function completely ignores the original input values $v_i$. Every single class gets assigned the exact same probability of $\frac{1}{n}$, giving you a perfectly uniform distribution.

In the context of LLMs, this means that if we reduce the temperature to exactly 0 (or almost 0), the model’s outputs—the new tokens—will become deterministic. Normally, models like GPT run at a higher default temperature, which is why you can input the same query twice and get different answers (same meaning, but different grammar or vocabulary). Lowering the temperature to 0 eliminates this variance. On the other hand, when we increase the temperature ($T \to +\infty$), the limit of the functions $S_T$ converges to the uniform distribution $S(v)_i=\frac{1}{n}$. This means that the model assigns the exact same weight to every word in the vocabulary, resulting in completely random and chaotic outputs.

To make these theoretical claims concrete, the following short demonstration shows what temperature actually looks like in practice on a real language model. The same prompt is submitted multiple times at progressively different temperature values.

This short demonstration illustrates the effect of the temperature parameter on the output of a locally hosted language model. In text generation, temperature is applied during the sampling stage after softmax, controlling how strongly the model favors high-probability tokens. Lower temperatures generally produce more deterministic and conservative outputs, while higher temperatures increase variability and can lead to more unexpected generations.

For the experiment, I used Ollama to run llama3.2:3b locally on my own machine. The command /set parameter temperature ... is used to modify the generation temperature, and /clear is used before each run to remove the previous conversation history and make the comparison cleaner. I also fix the seed to improve reproducibility, although the role of seed will be discussed separately in a later blog post.

Since the experiment is performed with a relatively small 3B model running locally, the results should be interpreted as a practical illustration rather than a general benchmark. The goal is to provide an intuitive example of how temperature influences language model behavior.

Interactive Exploration: Mapping Temperature in 3D

To really see how temperature shifts the model’s decision-making, let’s look at it geometrically. Since Softmax forces all output probabilities to sum to 1, we can visualize the model’s choices for a three-word vocabulary as a single point moving across a flat, triangular surface (mathematicians call a probability simplex). By watching how temperature drags this point around the triangle, we can see exactly how the model transitions from rigid confidence to total randomness. This triangle connects the absolute certainty points: $(1,0,0)$, $(0,1,0)$, and $(0,0,1)$.

Let’s fix a raw logit vector to observe how temperature moves the output probability across this space. In the following interactive plots, our model has output the raw scores:

\[v = [2.5, 1.0, -0.5]\]

Convergence to Argmax

In this first visualization, we lower the temperature from $T = 2.0$ down to near zero.

The Space: The $P_1$, $P_2$, and $P_3$ axes represent the probability assigned to each of our three logits. The gray outline is the simplex boundary.

The Trajectory: Follow the colored line. As $T$ decreases, the output distribution is aggressively pulled toward the corner vertex $(1, 0, 0)$. The model becomes 100% confident in $v_1$ (the maximum logit) and assigns 0% probability to the others.

Convergence to Uniform Distribution

Now, let’s see what happens when we heat things up. Using the exact same initial logit vector $v = [2.5, 1.0, -0.5]$, we increase the temperature from $T = 1.0$ up to $T = 50.0$.

The Trajectory: Instead of moving toward a corner, the rising temperature ignores the differences between the raw scores. The point is pulled directly into the center of the simplex (triangle), the coordinate $(1/3, 1/3, 1/3)$. At this center point, the model completely ignores the fact that $v_1$ was much larger than $v_3$. It assigns an equal $33.3\%$ probability to all three tokens, making the output completely random.

Truncation Sampling: Top-k and Top-p

After the Softmax function (with our chosen temperature) processes the logits, we are left with a valid probability distribution. But we are not done yet—we still need to actually pick a token from our vocabulary and add to the previous tokens (generative process).

If we always choose the token with the highest probability (a strategy called Greedy Search or Argmax), the output becomes highly repetitive, robotic, and boring.

For example, suppose the model has to complete the sentence:

The cat is sitting on the ___

After applying Softmax, the model may assign probabilities like this:

Token Probability
mat 0.50
sofa 0.25
floor 0.15
roof 0.09
moon 0.01

With greedy search, the model always chooses “mat”, because it has the highest probability. So every time we run the model in the same situation, we get the same completion:

The cat is sitting on the mat.

To achieve natural language and creativity, we need to treat the distribution like a more flexible object (which is one of the properties of distributions) and sample from it.

However, LLMs have vocabulary sizes in the tens of thousands (commonly 30k-60k). Even with temperature scaling, there is a “long tail” of thousands of irrelevant, ungrammatical, or nonsensical tokens that still possess a tiny fractional probability. If you generate a long essay, the model will eventually “roll” a bad number and pick one of these nonsense words. To prevent this, we use Truncation Sampling to cut off the tail before we sample.

Top-k Sampling :

Instead of considering the whole vocabulary, we sort the tokens by probability and only keep the top $K$ tokens (e.g., $K=50$). The probability of all other tokens is forced to $0$.

Top-p (Nucleus) Sampling:

Top-k is rigid; it keeps exactly $K$ tokens regardless of the model’s confidence. Top-p is dynamic. We sort the tokens by probability and keep adding them to a pool until their cumulative probability crosses a threshold $p$ (e.g., $p=0.90$). If the model is highly confident in 2 tokens, the pool is small. If the model is unsure and probabilities are flat, it might keep 100 tokens to reach the 90% threshold.

Re-normalization and Sampling

Because we discarded the tail in both Top-k and Top-p, the probabilities of our surviving tokens no longer sum to $1$. We must re-normalize them by dividing each surviving probability by the new total sum.

Finally, we randomly sample from this truncated, re-normalized distribution. This weighted random selection is the exact reason why submitting the identical query to an LLM multiple times will yield completely different, yet mathematically valid, responses!

Together, temperature scaling, truncation, and stochastic sampling form the inference-time pipeline that turns raw logits into a single next token and, when repeated, into a full completion. Every hyperparameter we have discussed so far is applied after the model has already been trained (meaning at inference time).

A natural question arises: if temperature is so powerful at controlling the model’s confidence, why is it only used at inference time? Why don’t we set $T$ as a trainable parameter and let gradient descent optimize it?

Why Don’t We Learn Temperature During Training?

During standard training, temperature is strictly set to $T = 1$. There are two main mathematical reasons for this: Scale Invariance and Gradient Instability.

Let’s look at the derivative of the Softmax function $S_T(v)_i$ with respect to a specific input logit $v_j$.

If we define $\delta_{ij}$ as the Kronecker delta (1 if $i = j$, and 0 otherwise), the gradient is:

\[\frac{\partial S_T(v)_i}{\partial v_j} = \frac{1}{T} S_T(v)_i (\delta_{ij} - S_T(v)_j)\]

Notice the $\frac{1}{T}$ multiplier at the front. If we were to allow $T$ to be a learnable parameter, we introduce severe instability to the backpropagation process. If the model pushes $T$ toward $0$, the gradients will explode (due to division by a tiny number). If $T$ grows large, the gradients vanish, halting learning entirely.

Furthermore, $T$ is mathematically redundant due to scale invariance. The logits $v$ are produced by the final linear layer: $v = Wx + b$. Because the temperature divides the logits ($\frac{v_i}{T}$), the model can achieve the exact same effect as changing $T$ by simply scaling its weights $W$ and biases $b$. By fixing $T=1$, we force the model to learn the actual absolute magnitudes of its weights to express confidence, keeping the optimization landscape identifiable and stable.

So far, we have looked at temperature mainly as a sampling control: a parameter that changes how sharply or softly the model chooses the next token. Lower temperature concentrates probability mass on the most likely tokens, while higher temperature spreads probability mass across more alternatives.

There is another way to look at the same effect: through Shannon entropy.

Shannon entropy measures how uncertain or spread out a probability distribution is. If almost all the probability is assigned to one token, the entropy is low. If the probability is distributed more evenly across many possible tokens, the entropy is higher.

In the following section we discuss a little bit about the relation between entropy and temperature.

This section is not essential for the main discussion, and the article can be understood without it. I include it only as a short side note, because entropy gives another natural language for describing the same idea of temperature: how concentrated or spread out a probability distribution is.

Information Theory: Temperature as an Entropy Dial

In the context of Information Theory, temperature acts as a dial that controls the Shannon Entropy of our output distribution.

Shannon Entropy ($H$) measures the unpredictability, surprise, or “chaos” inside a probability distribution $P = (p_1, p_2, \ldots, p_n)$. It is defined as:

\[H(P) = -\sum_{i=1}^n p_i \ln(p_i)\]

(Note: we assume $0 \ln(0) = 0$ based on the limit $\lim_{x \to 0^+} x \ln x = 0$).

When we use the temperature-scaled Softmax function to generate our probabilities ($p_i = S_T(v_i)$), changing $T$ directly manipulates the entropy of the system. Let’s look at our two extremes:

  1. As $T \to 0^+$ (The Deterministic Limit) We previously proved that as temperature approaches zero, the probability of the maximum logit approaches $1$, and all others approach $0$. Our distribution becomes a one-hot vector (e.g., $[1, 0, 0, \ldots, 0]$). Plugging this into the entropy formula gives:
\[H = -\left( 1 \ln(1) + 0 \ln(0) + \dots \right) = 0\]

At zero temperature, the system has zero entropy. The LLM is utterly deterministic; there is zero “surprise” in what it will output next.

  1. As $T \to \infty$ (The Chaotic Limit) Conversely, as temperature approaches infinity, every token gets the exact same probability: $p_i = \frac{1}{n}$. Plugging this uniform distribution into our formula:
\[H = -\sum_{i=1}^n \left( \frac{1}{n} \ln\left(\frac{1}{n}\right) \right) = -n \left( \frac{1}{n} \ln\left(\frac{1}{n}\right) \right) = \ln(n)\]

Here, $\ln(n)$ represents the absolute maximum possible entropy for a system with $n$ choices.

This behavior holds true for any arbitrary input vector $v$. By adjusting the temperature, we are shifting the machine (the LLM) anywhere between absolute certainty (0 entropy) and absolute randomness (maximum entropy).

Conclusion

We started with a simple question: how does a language model turn a vector of raw scores into an actual word? The answer required three pieces:

  • the Softmax function, which converts logits into a valid probability distribution;
  • temperature, which controls how concentrated or diffuse that distribution is;
  • and truncation sampling (Top-k, Top-p), which prunes the long tail before the final draw.

Along the way, we proved two limiting cases rigorously.

  • As $T \to 0^+$, the Softmax collapses pointwise to Argmax, the model becomes fully deterministic.
  • As $T \to \infty$, it converges to the uniform distribution , the model becomes fully random.

Every practical temperature setting lives somewhere between these two extremes, and the entropy framing makes this precise: temperature is a dial that moves the output distribution between zero entropy and $\ln(n)$.

One thing worth keeping in mind is that all of this happens after training. Temperature is not a learned parameter. Not because it is unimportant, but because it is mathematically redundant (the weights can absorb any constant scaling) and numerically dangerous to optimize (the gradient scales as $\frac{1}{T}$, which explodes near zero). Fixing $T = 1$ during training keeps the optimization landscape stable and identifiable.

The core takeaway is this: temperature does not change what the model knows , it changes how confidently it acts on what it knows.

References

  1. Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985). A Learning Algorithm for Boltzmann Machines. Cognitive Science, 9(1), 147–169. — Origin of temperature-based sampling from statistical mechanics.

  2. Bridle, J. S. (1990). Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition. In Neurocomputing (NATO ASI Series), Springer. — Introduces the Softmax function in its modern neural network form.

  3. Fan, A., Lewis, M., & Dauphin, Y. (2018). Hierarchical Neural Story Generation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), 889–898. arXiv:1805.04833 — Introduces Top-k sampling.

  4. Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The Curious Case of Neural Text Degeneration. International Conference on Learning Representations (ICLR). arXiv:1904.09751 — Introduces Nucleus (Top-p) Sampling.

  5. Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423. — Foundational definition of entropy used in the information-theoretic view of temperature.

  6. Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley-Interscience. — Standard reference for the formal properties of Shannon entropy used in the entropy-dial argument.

  7. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS), 30. arXiv:1706.03762 — Introduces the transformer architecture underlying decoder-only LLMs.

  8. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models Are Unsupervised Multitask Learners. OpenAI Technical Report (GPT-2). — Decoder-only LM using temperature and top-k sampling at inference time.

  9. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531 — Uses a raised softmax temperature to produce softened target distributions, an alternate use of temperature distinct from inference-time sampling.

  10. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. — General reference for Softmax, gradient backpropagation through non-differentiable functions like Argmax, and related fundamentals.