What is GPT?
- Machine Learning course
- Expert
- Lesson 23/44
- 5 min
GPT is an autoregressive text generation model.
What is GPT?
GPT (Generative Pre-trained Transformer) is a language model made only of transformer decoder blocks. Its pre-training task is a single one: predict the next token. No labels are needed; with a shift, the text itself becomes both input and target:
ids = [12, 7, 33, 5, 9]
giris = ids[:-1] # [12, 7, 33, 5]
hedef = ids[1:] # [7, 33, 5, 9]As scale grows, this simple task also brings out abilities like translation, summarization and writing code (GPT-2: 1.5B, GPT-3: 175B parameters).
Loss and perplexity
At each position the model gives a probability distribution over the vocabulary; the loss is the mean of the negative logarithm of the correct token's probability. Perplexity = e^loss; it tells among how many options on average the model is "undecided" at each step:
p_dogru = np.array([0.5, 0.25, 0.125])
kayip = -np.log(p_dogru).mean() # 1.386
print(np.exp(kayip)) # 4.0For a model guessing at random, perplexity equals the vocabulary size.
Practice Input, target and perplexity
Build the input and target sequences for the given ids and print them on one line. Then, for the model probabilities olasilik (each row a position), select the probabilities of the correct targets and print the loss and the perplexity with round(..., 3) on one line. Finally print the perplexity of a uniform distribution over an 8-word vocabulary.
Starter code
import numpy as np
ids = [3, 1, 4, 1, 5]
olasilik = np.array([[0.1, 0.5, 0.1, 0.1, 0.1, 0.1],
[0.05, 0.05, 0.05, 0.05, 0.75, 0.05],
[0.2, 0.4, 0.1, 0.1, 0.1, 0.1],
[0.1, 0.1, 0.1, 0.1, 0.1, 0.5]])
# buraya yazSample solution
import numpy as np
ids = [3, 1, 4, 1, 5]
olasilik = np.array([[0.1, 0.5, 0.1, 0.1, 0.1, 0.1],
[0.05, 0.05, 0.05, 0.05, 0.75, 0.05],
[0.2, 0.4, 0.1, 0.1, 0.1, 0.1],
[0.1, 0.1, 0.1, 0.1, 0.1, 0.5]])
giris, hedef = ids[:-1], ids[1:]
print(giris, hedef)
p = olasilik[np.arange(len(hedef)), hedef]
kayip = -np.log(p).mean()
print(round(kayip, 3), round(np.exp(kayip), 3))
print(round(np.exp(-np.log(np.full(8, 1 / 8)).mean()), 3))Expected output
[3, 1, 4, 1] [1, 4, 1, 5] 0.648 1.911 8.0
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why does GPT's pre-training not require labelled data?
- It only does classification
- There is no loss function
- The target is the next token of the text; the text is its own label Correct answer
- The model produces the labels
Explanation: This is called self-supervised learning.
2 Shift the target by one step.
hedef = ids[____:]hedef = ids[1:]Explanation: The target sequence starts at the second token.
3 Match.
| Term | Meaning |
|---|---|
ids[:-1] |
Input |
ids[1:] |
Target |
e^loss |
Perplexity |
uniform distribution |
Perplexity = vocabulary size |