Text generation
- Machine Learning course
- Expert
- Lesson 27/44
- 5 min
GPT generates text with greedy or beam search.
Text generation
At each step the model gives logits for the next token. Ways to pick a token from them:
- Greedy: the highest probability (
argmax); can produce repetitive, dull text. - Temperature (T): divide the logits by T and sample; T < 1 sharp, T > 1 creative/scattered.
- Top-k: sample only among the k most likely tokens.
- Top-p (nucleus): sample among the smallest set whose total probability reaches p.
p = softmax(logit / T)
sonraki = rng.choice(len(p), p=p)Chat models typically run around T ≈ 0.7 and top-p ≈ 0.9.
The effect of temperature
With the same logits, as the temperature changes the distribution sharpens or flattens:
logit = np.array([2.0, 1.0, 0.5, 0.0])
for T in [0.5, 1.0, 2.0]:
print(T, np.round(softmax(logit / T), 3))Top-k cuts off the tail of the distribution: logits outside k are set to −∞ and the rest are renormalized.
Practice Sampling strategies
For the given logit, print on separate lines: the greedy choice; the probabilities for T = 0.5 and T = 2 with np.round(..., 3); and the probabilities from a top_k(logit, 2) function (set everything outside k to −∞, apply softmax) with np.round(..., 3). Finally print the share of the first token in 10 000 samples at T = 1 with round(..., 2).
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
rng = np.random.default_rng(0)
logit = np.array([2.0, 1.0, 0.5, 0.0])
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
rng = np.random.default_rng(0)
logit = np.array([2.0, 1.0, 0.5, 0.0])
def top_k(logit, k):
z = np.full_like(logit, -np.inf)
en_iyi = np.argsort(logit)[-k:]
z[en_iyi] = logit[en_iyi]
return softmax(z)
print(int(np.argmax(logit)))
print(np.round(softmax(logit / 0.5), 3))
print(np.round(softmax(logit / 2), 3))
print(np.round(top_k(logit, 2), 3))
secim = rng.choice(4, size=10_000, p=softmax(logit))
print(round((secim == 0).mean(), 2))Expected output
0 [0.831 0.112 0.041 0.015] [0.409 0.248 0.193 0.15 ] [0.731 0.269 0. 0. ] 0.58
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 As the temperature approaches 0, what does sampling turn into?
- Top-p
- It raises an error
- Greedy selection (argmax) Correct answer
- Uniform random selection
Explanation: A small T magnifies differences; the probability piles onto the highest logit.
2 Scale the logits with the temperature.
p = softmax(logit ____ T)p = softmax(logit / T)Explanation: Dividing by T magnifies the differences when T < 1.
3 Match the methods.
| Term | Meaning |
|---|---|
argmax |
Greedy |
T > 1 |
Flattened distribution |
top-k |
The k most likely tokens |
top-p |
Probability mass p |