Summary - GPT

In this group you learned GPT, autoregressive generation, and causal masks.

Summary: GPT

In this group:

GPT is a decoder-only transformer; its single task is predicting the next token.
Input/target are obtained with a shift; the loss is cross-entropy, perplexity = e^loss.
Autoregressive generation rests on the chain rule and is sequential.
The causal mask sets everything above the diagonal to −∞; its correctness is tested with "the past doesn't change when the future changes".
Generation: greedy, temperature, top-k, top-p.

Exercise Generate text with a bigram

Build a bigram probability table from the given text (a matrix in sozluk order). Write the uret(baslangic, n, T) function: at each step apply the temperature to the last word's row with p ** (1 / T), renormalize and sample with rng.choice. First print the greedy (argmax at every step) 4-word continuation starting with "kedi", then a 6-word sample with T = 1 starting with "köpek".

Show example solution

import numpy as np

metin = "kedi süt içti . köpek su içti . kedi uyudu . köpek süt içti ."
kelimeler = metin.split()
sozluk = sorted(set(kelimeler))
idx = {k: i for i, k in enumerate(sozluk)}
rng = np.random.default_rng(1)

M = np.zeros((len(sozluk), len(sozluk)))
for a, b in zip(kelimeler, kelimeler[1:]):
    M[idx[a], idx[b]] += 1
M = M / M.sum(axis=1, keepdims=True)

def acgozlu(baslangic, n):
    dizi = [baslangic]
    for _ in range(n):
        dizi.append(sozluk[int(np.argmax(M[idx[dizi[-1]]]))])
    return " ".join(dizi)

def uret(baslangic, n, T=1.0):
    dizi = [baslangic]
    for _ in range(n):
        p = M[idx[dizi[-1]]] ** (1 / T)
        dizi.append(sozluk[rng.choice(len(p), p=p / p.sum())])
    return " ".join(dizi)

print(acgozlu("kedi", 4))
print(uret("köpek", 6))

Test yourself

Which suits GPT rather than BERT?
- Finding the word in a blank
- Writing the continuation of a story
- Classifying a sentence as positive/negative
- Entity recognition
Show answer

Writing the continuation of a story — Generation needs a left-to-right (causal) model.

Pick the next token greedily.

sonraki = int(np.____(logit))

Show answer

sonraki = int(np.argmax(logit))

Match.

ids[1:]

Target
triu(k=1)

Causal mask
T

Temperature
embedding.T

Weight tying

Next lesson: What is GAN?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies