CodeBlanks Logo Icon CodeBlanks
Login Register

Summary - GPT

In this group you learned GPT, autoregressive generation, and causal masks.

Summary: GPT

In this group:

  • GPT is a decoder-only transformer; its single task is predicting the next token.
  • Input/target are obtained with a shift; the loss is cross-entropy, perplexity = e^loss.
  • Autoregressive generation rests on the chain rule and is sequential.
  • The causal mask sets everything above the diagonal to −∞; its correctness is tested with "the past doesn't change when the future changes".
  • Generation: greedy, temperature, top-k, top-p.

Practice Generate text with a bigram

Build a bigram probability table from the given text (a matrix in sozluk order). Write the uret(baslangic, n, T) function: at each step apply the temperature to the last word's row with p ** (1 / T), renormalize and sample with rng.choice. First print the greedy (argmax at every step) 4-word continuation starting with "kedi", then a 6-word sample with T = 1 starting with "köpek".

Starter code

model.py
import numpy as np

metin = "kedi süt içti . köpek su içti . kedi uyudu . köpek süt içti ."
kelimeler = metin.split()
sozluk = sorted(set(kelimeler))
idx = {k: i for i, k in enumerate(sozluk)}
rng = np.random.default_rng(1)

# buraya yaz

Sample solution

model.py
import numpy as np

metin = "kedi süt içti . köpek su içti . kedi uyudu . köpek süt içti ."
kelimeler = metin.split()
sozluk = sorted(set(kelimeler))
idx = {k: i for i, k in enumerate(sozluk)}
rng = np.random.default_rng(1)

M = np.zeros((len(sozluk), len(sozluk)))
for a, b in zip(kelimeler, kelimeler[1:]):
    M[idx[a], idx[b]] += 1
M = M / M.sum(axis=1, keepdims=True)

def acgozlu(baslangic, n):
    dizi = [baslangic]
    for _ in range(n):
        dizi.append(sozluk[int(np.argmax(M[idx[dizi[-1]]]))])
    return " ".join(dizi)

def uret(baslangic, n, T=1.0):
    dizi = [baslangic]
    for _ in range(n):
        p = M[idx[dizi[-1]]] ** (1 / T)
        dizi.append(sozluk[rng.choice(len(p), p=p / p.sum())])
    return " ".join(dizi)

print(acgozlu("kedi", 4))
print(uret("köpek", 6))

Expected output

kedi süt içti . köpek
köpek süt içti . köpek su içti

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 Which suits GPT rather than BERT?

  • Finding the word in a blank
  • Writing the continuation of a story Correct answer
  • Classifying a sentence as positive/negative
  • Entity recognition

Explanation: Generation needs a left-to-right (causal) model.

2 Pick the next token greedily.

Question 
sonraki = int(np.____(logit))
Answer 
sonraki = int(np.argmax(logit))

Explanation: argmax gives the index of the highest logit.

3 Match.

Term Meaning
ids[1:] Target
triu(k=1) Causal mask
T Temperature
embedding.T Weight tying

All lessons in Expert

1 Attention

  1. What is attention?
  2. Self-attention
  3. Query, Key, Value
  4. Scaled dot-product
  5. Multi-head
  6. Summary - Attention

2 Transformer

  1. What is a transformer?
  2. Encoder
  3. Decoder
  4. Positional encoding
  5. TransformerEncoderLayer
  6. Summary - Transformer

3 Project: Transformer

  1. Preparing data
  2. Attention layer
  3. Encoder block
  4. Results

4 BERT

  1. What is BERT?
  2. Masked LM
  3. Tokenization
  4. Fine-tuning
  5. CLS token
  6. Summary - BERT

5 GPT

  1. What is GPT?
  2. Autoregressive
  3. Decoder-only
  4. Causal mask
  5. Text generation
  6. Summary - GPT

6 GAN and VAE

  1. What is GAN?
  2. Generator
  3. Discriminator
  4. What is VAE?
  5. Encoder-Decoder
  6. Summary - GAN and VAE

7 Reinforcement learning

  1. What is RL?
  2. Agent
  3. Policy
  4. Reward
  5. Q-learning
  6. Summary - RL

8 Expert final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4