Machine Learning
Causal mask
Causal mask lets GPT see only previous tokens.
The causal mask
In training the whole sequence is given at once; but position i's prediction must not see i+1, otherwise the model copies the answer. The causal mask sets everything above the diagonal to −∞:
n = 4
gelecek = np.triu(np.ones((n, n), dtype=bool), k=1)
skor[gelecek] = -np.inf
A = softmax(skor)
The result is a lower-triangular weight matrix: the first row looks only at itself, the last row at everyone.
Testing causality
In a correctly masked model, changing a future token does not change the output of earlier positions. This is the most reliable test for verifying an implementation:
y1 = model(x)
x2 = x.copy(); x2[-1] += 1
y2 = model(x2)
print(np.allclose(y1[:-1], y2[:-1])) # should be True
This property also makes it possible to cache the computation of earlier positions during generation (the KV cache).
Exercise Masked attention and the causality test
Write the nedensel_dikkat(x) function (Q = K = V = x, scaled, causally masked). Using the version that returns the weights, print the number of non-zero weights in each row. Then run the causality test: print on one line with np.allclose that changing the last row leaves the first three output rows unchanged and changes the last row.
Show example solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
x = np.random.default_rng(0).normal(size=(4, 6))
x2 = x.copy()
x2[-1] += 1
def nedensel_dikkat(x):
n, d = x.shape
skor = x @ x.T / np.sqrt(d)
skor[np.triu(np.ones((n, n), dtype=bool), k=1)] = -np.inf
A = softmax(skor)
return A @ x, A
Y, A = nedensel_dikkat(x)
print((A > 0).sum(axis=1))
Y2, _ = nedensel_dikkat(x2)
print(np.allclose(Y[:3], Y2[:3]), np.allclose(Y[3], Y2[3]))
Test yourself
-
Why is a GPT trained without a causal mask useless?
- The loss can't be computed
- Softmax becomes undefined
- The parameter count grows
- It sees and copies the next token directly; that information is absent at generation time
Show answer
It sees and copies the next token directly; that information is absent at generation time — Near-zero loss in training, but meaningless output at generation.
-
Select above the diagonal.
gelecek = np.triu(np.ones((n, n), dtype=bool), k=____)
Show answer
gelecek = np.triu(np.ones((n, n), dtype=bool), k=1)
-
Match (n=4).
- A[0]
- Looks only at itself
- A[3]
- Looks at four positions
- triu(k=1)
- Future positions
- KV cache
- Stores earlier computation
Next lesson: Text generation
