Machine Learning
Decoder
The decoder uses encoder output and previous outputs to generate predictions.
The decoder
The decoder produces the output sequence from left to right. It has two differences from the encoder block:
- Masked self-attention: position i only looks at 0..i; it cannot see future words.
- Cross-attention: the Query comes from the decoder, the Key and Value from the encoder output; in translation the target word looks at the source sentence.
x = layer_norm(x + masked_self_attn(x))
x = layer_norm(x + cross_attn(Q=x, K=enc, V=enc))
x = layer_norm(x + ffn(x))
The cross-attention weight matrix has shape (target length × source length); it need not be square.
The causal mask
Before softmax, the mask sets the scores of future positions to −∞, so their weights become exactly 0. It is built with a lower-triangular boolean matrix:
n = 4
maske = np.tril(np.ones((n, n), dtype=bool))
skor = np.where(maske, skor, -np.inf)
A = softmax(skor, eksen=-1)
print(np.round(A[1], 2)) # only the first two positions are above zero
In training the whole target sequence is given at once; thanks to the mask each position still sees only the past (teacher forcing).
Exercise Masked and cross-attention
Compute the masked self-attention weights for the decoder sequence y (3 positions) and print them with np.round(..., 3). Then compute the cross-attention weights with the encoder output enc (5 positions) and print only their shape.
Show example solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
y = np.array([[1., 0.], [0., 1.], [1., 1.]])
enc = np.random.default_rng(0).normal(size=(5, 2))
n = len(y)
maske = np.tril(np.ones((n, n), dtype=bool))
skor = np.where(maske, y @ y.T / np.sqrt(2), -np.inf)
print(np.round(softmax(skor), 3))
C = softmax(y @ enc.T / np.sqrt(2))
print(C.shape)
Test yourself
-
Where do the Key and Value come from in cross-attention?
- From the decoder itself
- From the positional encoding
- They are randomly initialized
- From the encoder output
Show answer
From the encoder output — The Query comes from the decoder, Key/Value from the encoder.
-
Mask the future positions.
skor = np.where(maske, skor, -np.____)
Show answer
skor = np.where(maske, skor, -np.inf)
-
Match the attention types.
- encoder self-attention
- Looks at the whole sequence
- masked self-attention
- Looks only at the past
- cross-attention
- Looks at the source sentence
- np.tril
- Lower-triangular mask
Next lesson: Positional encoding
