Machine Learning
What is GPT?
GPT is an autoregressive text generation model.
What is GPT?
GPT (Generative Pre-trained Transformer) is a language model made only of transformer decoder blocks. Its pre-training task is a single one: predict the next token. No labels are needed; with a shift, the text itself becomes both input and target:
ids = [12, 7, 33, 5, 9]
giris = ids[:-1] # [12, 7, 33, 5]
hedef = ids[1:] # [7, 33, 5, 9]
As scale grows, this simple task also brings out abilities like translation, summarization and writing code (GPT-2: 1.5B, GPT-3: 175B parameters).
Loss and perplexity
At each position the model gives a probability distribution over the vocabulary; the loss is the mean of the negative logarithm of the correct token's probability. Perplexity = e^loss; it tells among how many options on average the model is "undecided" at each step:
p_dogru = np.array([0.5, 0.25, 0.125])
kayip = -np.log(p_dogru).mean() # 1.386
print(np.exp(kayip)) # 4.0
For a model guessing at random, perplexity equals the vocabulary size.
Exercise Input, target and perplexity
Build the input and target sequences for the given ids and print them on one line. Then, for the model probabilities olasilik (each row a position), select the probabilities of the correct targets and print the loss and the perplexity with round(..., 3) on one line. Finally print the perplexity of a uniform distribution over an 8-word vocabulary.
Show example solution
import numpy as np
ids = [3, 1, 4, 1, 5]
olasilik = np.array([[0.1, 0.5, 0.1, 0.1, 0.1, 0.1],
[0.05, 0.05, 0.05, 0.05, 0.75, 0.05],
[0.2, 0.4, 0.1, 0.1, 0.1, 0.1],
[0.1, 0.1, 0.1, 0.1, 0.1, 0.5]])
giris, hedef = ids[:-1], ids[1:]
print(giris, hedef)
p = olasilik[np.arange(len(hedef)), hedef]
kayip = -np.log(p).mean()
print(round(kayip, 3), round(np.exp(kayip), 3))
print(round(np.exp(-np.log(np.full(8, 1 / 8)).mean()), 3))
Test yourself
-
Why does GPT's pre-training not require labelled data?
- It only does classification
- There is no loss function
- The target is the next token of the text; the text is its own label
- The model produces the labels
Show answer
The target is the next token of the text; the text is its own label — This is called self-supervised learning.
-
Shift the target by one step.
hedef = ids[____:]
Show answer
hedef = ids[1:]
-
Match.
- ids[:-1]
- Input
- ids[1:]
- Target
- e^loss
- Perplexity
- uniform distribution
- Perplexity = vocabulary size
Next lesson: Autoregressive
