What is a transformer?
- Machine Learning course
- Expert
- Lesson 7/44
- 5 min
A transformer is a sequence model based on the attention mechanism.
What is a transformer?
The transformer (2017, "Attention Is All You Need") is a sequence model built only from attention and feed-forward layers, without recurrent layers. A block consists of two sub-layers; each is wrapped in a residual connection and layer normalization:
# one encoder block
x = layer_norm(x + self_attention(x))
x = layer_norm(x + ffn(x)) # ffn: Linear -> ReLU -> LinearBlocks are stacked (BERT-base: 12, GPT-3: 96). Since the input and output shape always stays (n, d_model), stacking is easy.
Residual connection and LayerNorm
The residual connection (x + f(x)) lets the gradient flow directly through a deep network. LayerNorm normalizes each position's vector within itself (mean 0, variance 1); unlike BatchNorm it is independent of the batch size:
def layer_norm(x, eps=1e-5):
m = x.mean(axis=-1, keepdims=True)
v = x.var(axis=-1, keepdims=True)
return (x - m) / np.sqrt(v + eps)
print(layer_norm(np.array([1., 2., 3., 4.]))) # [-1.342 -0.447 0.447 1.342]The real LayerNorm also applies a learned scale (γ) and shift (β) after normalizing.
Practice LayerNorm
Write the layer_norm function. For the given x (3 positions × 4 dimensions), print each row's mean and standard deviation of the result with np.round(..., 3) on separate lines. Then print the normalized first row with np.round(..., 3).
Starter code
import numpy as np
x = np.array([[1., 2., 3., 4.], [10., 10., 20., 20.], [-1., 0., 0., 1.]])
# buraya yazSample solution
import numpy as np
x = np.array([[1., 2., 3., 4.], [10., 10., 20., 20.], [-1., 0., 0., 1.]])
def layer_norm(x, eps=1e-5):
m = x.mean(axis=-1, keepdims=True)
v = x.var(axis=-1, keepdims=True)
return (x - m) / np.sqrt(v + eps)
y = layer_norm(x)
print(np.round(y.mean(axis=-1), 3))
print(np.round(y.std(axis=-1), 3))
print(np.round(y[0], 3))Expected output
[0. 0. 0.] [1. 1. 1.] [-1.342 -0.447 0.447 1.342]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What makes it easy to stack transformer blocks?
- Pooling layers
- The input and output shape staying the same (n, d_model) Correct answer
- Parameter sharing
- Recurrent connections
Explanation: Each block takes and gives the same shape.
2 Add the residual connection.
x = layer_norm(x ____ self_attention(x))x = layer_norm(x + self_attention(x))Explanation: The residual connection adds the input to the sub-layer's output.
3 Match the parts.
| Term | Meaning |
|---|---|
self-attention |
Information across positions |
FFN |
Per-position transformation |
x + f(x) |
Residual connection |
LayerNorm |
Per-vector normalization |