Machine Learning
What is a transformer?
A transformer is a sequence model based on the attention mechanism.
What is a transformer?
The transformer (2017, "Attention Is All You Need") is a sequence model built only from attention and feed-forward layers, without recurrent layers. A block consists of two sub-layers; each is wrapped in a residual connection and layer normalization:
# one encoder block
x = layer_norm(x + self_attention(x))
x = layer_norm(x + ffn(x)) # ffn: Linear -> ReLU -> Linear
Blocks are stacked (BERT-base: 12, GPT-3: 96). Since the input and output shape always stays (n, d_model), stacking is easy.
Residual connection and LayerNorm
The residual connection (x + f(x)) lets the gradient flow directly through a deep network. LayerNorm normalizes each position's vector within itself (mean 0, variance 1); unlike BatchNorm it is independent of the batch size:
def layer_norm(x, eps=1e-5):
m = x.mean(axis=-1, keepdims=True)
v = x.var(axis=-1, keepdims=True)
return (x - m) / np.sqrt(v + eps)
print(layer_norm(np.array([1., 2., 3., 4.]))) # [-1.342 -0.447 0.447 1.342]
The real LayerNorm also applies a learned scale (γ) and shift (β) after normalizing.
Exercise LayerNorm
Write the layer_norm function. For the given x (3 positions × 4 dimensions), print each row's mean and standard deviation of the result with np.round(..., 3) on separate lines. Then print the normalized first row with np.round(..., 3).
Show example solution
import numpy as np
x = np.array([[1., 2., 3., 4.], [10., 10., 20., 20.], [-1., 0., 0., 1.]])
def layer_norm(x, eps=1e-5):
m = x.mean(axis=-1, keepdims=True)
v = x.var(axis=-1, keepdims=True)
return (x - m) / np.sqrt(v + eps)
y = layer_norm(x)
print(np.round(y.mean(axis=-1), 3))
print(np.round(y.std(axis=-1), 3))
print(np.round(y[0], 3))
Test yourself
-
What makes it easy to stack transformer blocks?
- Pooling layers
- The input and output shape staying the same (n, d_model)
- Parameter sharing
- Recurrent connections
Show answer
The input and output shape staying the same (n, d_model) — Each block takes and gives the same shape.
-
Add the residual connection.
x = layer_norm(x ____ self_attention(x))
Show answer
x = layer_norm(x + self_attention(x))
-
Match the parts.
- self-attention
- Information across positions
- FFN
- Per-position transformation
- x + f(x)
- Residual connection
- LayerNorm
- Per-vector normalization
Next lesson: Encoder
