Machine Learning
Autoregressive
GPT generates text by predicting the next token from previous ones.
The autoregressive model
An autoregressive model factorizes a sequence's probability with the chain rule: each token is conditioned on the ones before it.
P(kedi süt içti) = P(kedi) · P(süt | kedi) · P(içti | kedi süt)
Generation follows the same path: pick a token, append it to the input, repeat. That is why generation is sequential; producing 100 tokens means running the model 100 times (the KV cache speeds up this repetition).
The simplest language model: bigram
If we limit the context to only the previous word we get a bigram model: P(next | previous) = count(previous, next) / count(previous). GPT is the version of this that extends the context to thousands of tokens and uses a learned network instead of counts:
from collections import Counter
ciftler = Counter(zip(kelimeler, kelimeler[1:]))
once = Counter(kelimeler[:-1])
p = ciftler[("kedi", "süt")] / once["kedi"] # 0.5
A bigram model can also produce "köpek süt içti", but it can't capture long-range relationships.
Exercise A bigram language model
Extract the bigram counts from the given text and write the p(onceki, sonraki) function. Print p("kedi", "süt") and p("süt", "içti") on one line. Then print the probability of the sequence "kedi süt içti ." with the chain rule (assuming it starts with "kedi") with round(..., 3).
Show example solution
from collections import Counter
metin = "kedi süt içti . köpek su içti . kedi uyudu . köpek süt içti ."
kelimeler = metin.split()
ciftler = Counter(zip(kelimeler, kelimeler[1:]))
once = Counter(kelimeler[:-1])
def p(onceki, sonraki):
return ciftler[(onceki, sonraki)] / once[onceki]
print(p("kedi", "süt"), p("süt", "içti"))
dizi = "kedi süt içti .".split()
olas = 1.0
for a, b in zip(dizi, dizi[1:]):
olas *= p(a, b)
print(round(olas, 3))
Test yourself
-
In autoregressive generation, how many forward passes does the model make to produce 50 tokens?
Show answer
50 — Each new token requires one forward pass.
-
Count consecutive word pairs.
ciftler = Counter(____(kelimeler, kelimeler[1:]))
Show answer
ciftler = Counter(zip(kelimeler, kelimeler[1:]))
-
Match the bigram probabilities.
- P(süt | kedi)
- 0.5
- P(içti | su)
- 1.0
- P(kedi | .)
- ≈ 0.33
- P(uyudu | süt)
- 0.0
Next lesson: Decoder-only
