Summary - Transformer
- Machine Learning course
- Expert
- Lesson 12/44
- 6 min
In this group you learned encoder, decoder, and positional encoding.
Summary: transformer
In this group:
- Transformer block: self-attention + FFN, each with a residual connection and LayerNorm.
- The encoder sees the whole sequence; the FFN is applied per position.
- The decoder uses a causal mask and cross-attention (Q from the decoder, K/V from the encoder).
- Attention is unaware of order; a positional encoding (sinusoidal, learned or RoPE) is added.
- In PyTorch,
nn.TransformerEncoderLayerandnn.TransformerEncodercome ready-made.
Practice A mini transformer block
Write a blok(x) function combining the positional code, causally masked attention, and residual + LayerNorm: first x + konum_kodu, then masked self-attention, then layer_norm(x + attention). Print the output shape. Then test causality: print with np.allclose that changing the last row of x leaves the output of the first three rows unchanged.
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
def layer_norm(x, eps=1e-5):
m = x.mean(axis=-1, keepdims=True)
v = x.var(axis=-1, keepdims=True)
return (x - m) / np.sqrt(v + eps)
def konum_kodu(n, d):
pos = np.arange(n)[:, None]
aci = pos / 10000 ** (np.arange(0, d, 2) / d)
PE = np.zeros((n, d))
PE[:, 0::2], PE[:, 1::2] = np.sin(aci), np.cos(aci)
return PE
x = np.random.default_rng(0).normal(size=(4, 6))
x2 = x.copy()
x2[-1] += 3
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
def layer_norm(x, eps=1e-5):
m = x.mean(axis=-1, keepdims=True)
v = x.var(axis=-1, keepdims=True)
return (x - m) / np.sqrt(v + eps)
def konum_kodu(n, d):
pos = np.arange(n)[:, None]
aci = pos / 10000 ** (np.arange(0, d, 2) / d)
PE = np.zeros((n, d))
PE[:, 0::2], PE[:, 1::2] = np.sin(aci), np.cos(aci)
return PE
x = np.random.default_rng(0).normal(size=(4, 6))
x2 = x.copy()
x2[-1] += 3
def blok(x):
n, d = x.shape
x = x + konum_kodu(n, d)
maske = np.tril(np.ones((n, n), dtype=bool))
skor = np.where(maske, x @ x.T / np.sqrt(d), -np.inf)
return layer_norm(x + softmax(skor) @ x)
print(blok(x).shape)
print(np.allclose(blok(x)[:3], blok(x2)[:3]))Expected output
(4, 6) True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 In a translation model, which part connects the source sentence to the target word?
- Cross-attention Correct answer
- Positional encoding
- LayerNorm
- FFN
Explanation: In cross-attention the target's query looks at the source's keys.
2 Build the causal mask.
Question
maske = np.____(np.ones((n, n), dtype=bool))
Answer
maske = np.tril(np.ones((n, n), dtype=bool))Explanation: Lower triangle: position i sees 0..i.
3 Match.
| Term | Meaning |
|---|---|
encoder |
Bidirectional context |
decoder |
Left-to-right generation |
positional encoding |
Order information |
residual connection |
Gradient highway |