Autoregressive
- Machine Learning course
- Expert
- Lesson 24/44
- 5 min
GPT generates text by predicting the next token from previous ones.
The autoregressive model
An autoregressive model factorizes a sequence's probability with the chain rule: each token is conditioned on the ones before it.
P(kedi süt içti) = P(kedi) · P(süt | kedi) · P(içti | kedi süt)Generation follows the same path: pick a token, append it to the input, repeat. That is why generation is sequential; producing 100 tokens means running the model 100 times (the KV cache speeds up this repetition).
The simplest language model: bigram
If we limit the context to only the previous word we get a bigram model: P(next | previous) = count(previous, next) / count(previous). GPT is the version of this that extends the context to thousands of tokens and uses a learned network instead of counts:
from collections import Counter
ciftler = Counter(zip(kelimeler, kelimeler[1:]))
once = Counter(kelimeler[:-1])
p = ciftler[("kedi", "süt")] / once["kedi"] # 0.5A bigram model can also produce "köpek süt içti", but it can't capture long-range relationships.
Practice A bigram language model
Extract the bigram counts from the given text and write the p(onceki, sonraki) function. Print p("kedi", "süt") and p("süt", "içti") on one line. Then print the probability of the sequence "kedi süt içti ." with the chain rule (assuming it starts with "kedi") with round(..., 3).
Starter code
from collections import Counter
metin = "kedi süt içti . köpek su içti . kedi uyudu . köpek süt içti ."
kelimeler = metin.split()
# buraya yazSample solution
from collections import Counter
metin = "kedi süt içti . köpek su içti . kedi uyudu . köpek süt içti ."
kelimeler = metin.split()
ciftler = Counter(zip(kelimeler, kelimeler[1:]))
once = Counter(kelimeler[:-1])
def p(onceki, sonraki):
return ciftler[(onceki, sonraki)] / once[onceki]
print(p("kedi", "süt"), p("süt", "içti"))
dizi = "kedi süt içti .".split()
olas = 1.0
for a, b in zip(dizi, dizi[1:]):
olas *= p(a, b)
print(round(olas, 3))Expected output
0.5 1.0 0.5
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 In autoregressive generation, how many forward passes does the model make to produce 50 tokens?
- 2500
- 50 Correct answer
- 1
- 25
Explanation: Each new token requires one forward pass.
2 Count consecutive word pairs.
ciftler = Counter(____(kelimeler, kelimeler[1:]))ciftler = Counter(zip(kelimeler, kelimeler[1:]))Explanation: zip pairs each word with the next.
3 Match the bigram probabilities.
| Term | Meaning |
|---|---|
P(süt | kedi) |
0.5 |
P(içti | su) |
1.0 |
P(kedi | .) |
≈ 0.33 |
P(uyudu | süt) |
0.0 |