Causal mask
- Machine Learning course
- Expert
- Lesson 26/44
- 5 min
Causal mask lets GPT see only previous tokens.
The causal mask
In training the whole sequence is given at once; but position i's prediction must not see i+1, otherwise the model copies the answer. The causal mask sets everything above the diagonal to −∞:
n = 4
gelecek = np.triu(np.ones((n, n), dtype=bool), k=1)
skor[gelecek] = -np.inf
A = softmax(skor)The result is a lower-triangular weight matrix: the first row looks only at itself, the last row at everyone.
Testing causality
In a correctly masked model, changing a future token does not change the output of earlier positions. This is the most reliable test for verifying an implementation:
y1 = model(x)
x2 = x.copy(); x2[-1] += 1
y2 = model(x2)
print(np.allclose(y1[:-1], y2[:-1])) # should be TrueThis property also makes it possible to cache the computation of earlier positions during generation (the KV cache).
Practice Masked attention and the causality test
Write the nedensel_dikkat(x) function (Q = K = V = x, scaled, causally masked). Using the version that returns the weights, print the number of non-zero weights in each row. Then run the causality test: print on one line with np.allclose that changing the last row leaves the first three output rows unchanged and changes the last row.
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
x = np.random.default_rng(0).normal(size=(4, 6))
x2 = x.copy()
x2[-1] += 1
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
x = np.random.default_rng(0).normal(size=(4, 6))
x2 = x.copy()
x2[-1] += 1
def nedensel_dikkat(x):
n, d = x.shape
skor = x @ x.T / np.sqrt(d)
skor[np.triu(np.ones((n, n), dtype=bool), k=1)] = -np.inf
A = softmax(skor)
return A @ x, A
Y, A = nedensel_dikkat(x)
print((A > 0).sum(axis=1))
Y2, _ = nedensel_dikkat(x2)
print(np.allclose(Y[:3], Y2[:3]), np.allclose(Y[3], Y2[3]))Expected output
[1 2 3 4] True False
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why is a GPT trained without a causal mask useless?
- The loss can't be computed
- Softmax becomes undefined
- The parameter count grows
- It sees and copies the next token directly; that information is absent at generation time Correct answer
Explanation: Near-zero loss in training, but meaningless output at generation.
2 Select above the diagonal.
gelecek = np.triu(np.ones((n, n), dtype=bool), k=____)gelecek = np.triu(np.ones((n, n), dtype=bool), k=1)Explanation: k=1 excludes the diagonal; a position can see itself.
3 Match (n=4).
| Term | Meaning |
|---|---|
A[0] |
Looks only at itself |
A[3] |
Looks at four positions |
triu(k=1) |
Future positions |
KV cache |
Stores earlier computation |