Machine Learning
Text generation
GPT generates text with greedy or beam search.
Text generation
At each step the model gives logits for the next token. Ways to pick a token from them:
- Greedy: the highest probability (
argmax); can produce repetitive, dull text. - Temperature (T): divide the logits by T and sample; T < 1 sharp, T > 1 creative/scattered.
- Top-k: sample only among the k most likely tokens.
- Top-p (nucleus): sample among the smallest set whose total probability reaches p.
p = softmax(logit / T)
sonraki = rng.choice(len(p), p=p)
Chat models typically run around T ≈ 0.7 and top-p ≈ 0.9.
The effect of temperature
With the same logits, as the temperature changes the distribution sharpens or flattens:
logit = np.array([2.0, 1.0, 0.5, 0.0])
for T in [0.5, 1.0, 2.0]:
print(T, np.round(softmax(logit / T), 3))
Top-k cuts off the tail of the distribution: logits outside k are set to −∞ and the rest are renormalized.
Exercise Sampling strategies
For the given logit, print on separate lines: the greedy choice; the probabilities for T = 0.5 and T = 2 with np.round(..., 3); and the probabilities from a top_k(logit, 2) function (set everything outside k to −∞, apply softmax) with np.round(..., 3). Finally print the share of the first token in 10 000 samples at T = 1 with round(..., 2).
Show example solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
rng = np.random.default_rng(0)
logit = np.array([2.0, 1.0, 0.5, 0.0])
def top_k(logit, k):
z = np.full_like(logit, -np.inf)
en_iyi = np.argsort(logit)[-k:]
z[en_iyi] = logit[en_iyi]
return softmax(z)
print(int(np.argmax(logit)))
print(np.round(softmax(logit / 0.5), 3))
print(np.round(softmax(logit / 2), 3))
print(np.round(top_k(logit, 2), 3))
secim = rng.choice(4, size=10_000, p=softmax(logit))
print(round((secim == 0).mean(), 2))
Test yourself
-
As the temperature approaches 0, what does sampling turn into?
- Top-p
- It raises an error
- Greedy selection (argmax)
- Uniform random selection
Show answer
Greedy selection (argmax) — A small T magnifies differences; the probability piles onto the highest logit.
-
Scale the logits with the temperature.
p = softmax(logit ____ T)
Show answer
p = softmax(logit / T)
-
Match the methods.
- argmax
- Greedy
- T > 1
- Flattened distribution
- top-k
- The k most likely tokens
- top-p
- Probability mass p
Next lesson: Summary - GPT
