Machine Learning
Self-attention
Self-attention computes the relationship of each element with all others.
Self-attention
In self-attention every element of a sequence attends to all elements of the same sequence (including itself). For a sequence of n elements an n × n weight matrix forms; each row is one element's distribution:
X = np.random.default_rng(0).normal(size=(4, 8)) # 4 elements, 8 dimensions
skor = X @ X.T # (4, 4) similarities
A = softmax(skor, eksen=1) # probability per row
Y = A @ X # (4, 8) new representations
The computation grows with n²: a 1000-word text means a matrix of a million. This is the core cost of long-context models.
Properties of the attention matrix
Because softmax is applied per row, every row sums to 1 and all elements are positive. Since similarity is highest with itself (a vector's dot product with itself), the diagonal is usually dominant:
print(A.sum(axis=1)) # [1. 1. 1. 1.]
print((A.argmax(axis=1) == np.arange(4)).all()) # each element looks at itself most
In real models Q, K and V are separately learned transformations, so this diagonal dominance is broken; you'll see this in the next lesson.
Exercise Self-attention
For the given X, compute the attention matrix from the X @ X.T scores with a per-row softmax. Print its shape and the row sums (np.round(..., 3)) on separate lines, then that every element attends most to itself (argmax on the diagonal).
Show example solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
X = np.random.default_rng(0).normal(size=(4, 8))
A = softmax(X @ X.T, eksen=1)
print(A.shape)
print(np.round(A.sum(axis=1), 3))
print((A.argmax(axis=1) == np.arange(4)).all())
Test yourself
-
How many elements does the attention matrix have for a 512-token sequence?
Show answer
262144 — 512 × 512 = 262144; doubling the length quadruples the cost.
-
Compute the similarity of all pairs.
skor = X @ X.____
Show answer
skor = X @ X.T
-
Match the shapes (n=4, d=8).
- X
- (4, 8)
- X @ X.T
- (4, 4)
- A.sum(axis=1)
- All ones
- A[0]
- (4,)
Next lesson: Query, Key, Value
