Decoder
- Machine Learning course
- Expert
- Lesson 9/44
- 5 min
The decoder uses encoder output and previous outputs to generate predictions.
The decoder
The decoder produces the output sequence from left to right. It has two differences from the encoder block:
- Masked self-attention: position i only looks at 0..i; it cannot see future words.
- Cross-attention: the Query comes from the decoder, the Key and Value from the encoder output; in translation the target word looks at the source sentence.
x = layer_norm(x + masked_self_attn(x))
x = layer_norm(x + cross_attn(Q=x, K=enc, V=enc))
x = layer_norm(x + ffn(x))The cross-attention weight matrix has shape (target length × source length); it need not be square.
The causal mask
Before softmax, the mask sets the scores of future positions to −∞, so their weights become exactly 0. It is built with a lower-triangular boolean matrix:
n = 4
maske = np.tril(np.ones((n, n), dtype=bool))
skor = np.where(maske, skor, -np.inf)
A = softmax(skor, eksen=-1)
print(np.round(A[1], 2)) # only the first two positions are above zeroIn training the whole target sequence is given at once; thanks to the mask each position still sees only the past (teacher forcing).
Practice Masked and cross-attention
Compute the masked self-attention weights for the decoder sequence y (3 positions) and print them with np.round(..., 3). Then compute the cross-attention weights with the encoder output enc (5 positions) and print only their shape.
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
y = np.array([[1., 0.], [0., 1.], [1., 1.]])
enc = np.random.default_rng(0).normal(size=(5, 2))
n = len(y)
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
y = np.array([[1., 0.], [0., 1.], [1., 1.]])
enc = np.random.default_rng(0).normal(size=(5, 2))
n = len(y)
maske = np.tril(np.ones((n, n), dtype=bool))
skor = np.where(maske, y @ y.T / np.sqrt(2), -np.inf)
print(np.round(softmax(skor), 3))
C = softmax(y @ enc.T / np.sqrt(2))
print(C.shape)Expected output
[[1. 0. 0. ] [0.33 0.67 0. ] [0.248 0.248 0.503]] (3, 5)
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Where do the Key and Value come from in cross-attention?
- From the decoder itself
- From the positional encoding
- They are randomly initialized
- From the encoder output Correct answer
Explanation: The Query comes from the decoder, Key/Value from the encoder.
2 Mask the future positions.
skor = np.where(maske, skor, -np.____)skor = np.where(maske, skor, -np.inf)Explanation: The exponential of −∞ is 0; the weight becomes exactly zero.
3 Match the attention types.
| Term | Meaning |
|---|---|
encoder self-attention |
Looks at the whole sequence |
masked self-attention |
Looks only at the past |
cross-attention |
Looks at the source sentence |
np.tril |
Lower-triangular mask |