Summary - Transformer

In this group you learned encoder, decoder, and positional encoding.

Summary: transformer

In this group:

Transformer block: self-attention + FFN, each with a residual connection and LayerNorm.
The encoder sees the whole sequence; the FFN is applied per position.
The decoder uses a causal mask and cross-attention (Q from the decoder, K/V from the encoder).
Attention is unaware of order; a positional encoding (sinusoidal, learned or RoPE) is added.
In PyTorch, nn.TransformerEncoderLayer and nn.TransformerEncoder come ready-made.

Exercise A mini transformer block

Write a blok(x) function combining the positional code, causally masked attention, and residual + LayerNorm: first x + konum_kodu, then masked self-attention, then layer_norm(x + attention). Print the output shape. Then test causality: print with np.allclose that changing the last row of x leaves the output of the first three rows unchanged.

Show example solution

import numpy as np

def softmax(z, eksen=-1):
    e = np.exp(z - z.max(axis=eksen, keepdims=True))
    return e / e.sum(axis=eksen, keepdims=True)

def layer_norm(x, eps=1e-5):
    m = x.mean(axis=-1, keepdims=True)
    v = x.var(axis=-1, keepdims=True)
    return (x - m) / np.sqrt(v + eps)

def konum_kodu(n, d):
    pos = np.arange(n)[:, None]
    aci = pos / 10000 ** (np.arange(0, d, 2) / d)
    PE = np.zeros((n, d))
    PE[:, 0::2], PE[:, 1::2] = np.sin(aci), np.cos(aci)
    return PE

x = np.random.default_rng(0).normal(size=(4, 6))
x2 = x.copy()
x2[-1] += 3

def blok(x):
    n, d = x.shape
    x = x + konum_kodu(n, d)
    maske = np.tril(np.ones((n, n), dtype=bool))
    skor = np.where(maske, x @ x.T / np.sqrt(d), -np.inf)
    return layer_norm(x + softmax(skor) @ x)

print(blok(x).shape)
print(np.allclose(blok(x)[:3], blok(x2)[:3]))

Test yourself

In a translation model, which part connects the source sentence to the target word?
- Cross-attention
- Positional encoding
- LayerNorm
- FFN
Show answer

Cross-attention — In cross-attention the target's query looks at the source's keys.

Build the causal mask.

maske = np.____(np.ones((n, n), dtype=bool))

Show answer

maske = np.tril(np.ones((n, n), dtype=bool))

Match.

encoder

Bidirectional context
decoder

Left-to-right generation
positional encoding

Order information
residual connection

Gradient highway

Next lesson: Preparing data

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies