Machine Learning
Positional encoding
Positional encoding gives the model position information in the sequence.
Positional encoding
Attention is unaware of position: if you shuffle the order of the words, the outputs are shuffled the same way (permutation-equivariant). To convey order, a positional encoding is added to the input. The original paper uses sine/cosine:
def konum_kodu(n, d):
pos = np.arange(n)[:, None]
i = np.arange(0, d, 2)
aci = pos / 10000 ** (i / d)
PE = np.zeros((n, d))
PE[:, 0::2] = np.sin(aci)
PE[:, 1::2] = np.cos(aci)
return PE
x = gomme + konum_kodu(n, d)
Low dimensions oscillate fast and high dimensions slowly; this way every position gets a unique "fingerprint".
Permutation equivariance
You can see this numerically: reversing the rows and applying self-attention is the same as reversing self-attention's output. Once the positional encoding is added, this equality breaks:
ters = x[::-1]
print(np.allclose(oz_dikkat(ters), oz_dikkat(x)[::-1])) # True
xp = x + konum_kodu(n, d)
tp = x[::-1] + konum_kodu(n, d)
print(np.allclose(oz_dikkat(tp), oz_dikkat(xp)[::-1])) # False
Models like BERT and GPT use learned position embeddings instead of fixed sines; newer models prefer relative methods such as RoPE.
Exercise Sinusoidal positional code
Write the konum_kodu(n, d) function. Print the shape and first row of konum_kodu(4, 6) on separate lines, then its second row with np.round(..., 3). Finally print on one line with np.allclose that, with the given oz_dikkat, reversing the rows is equivariant without the positional code and not equivariant with it.
Show example solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
def oz_dikkat(x):
return softmax(x @ x.T / np.sqrt(x.shape[1])) @ x
x = np.random.default_rng(0).normal(size=(4, 6))
def konum_kodu(n, d):
pos = np.arange(n)[:, None]
i = np.arange(0, d, 2)
aci = pos / 10000 ** (i / d)
PE = np.zeros((n, d))
PE[:, 0::2] = np.sin(aci)
PE[:, 1::2] = np.cos(aci)
return PE
PE = konum_kodu(4, 6)
print(PE.shape)
print(PE[0])
print(np.round(PE[1], 3))
esit1 = np.allclose(oz_dikkat(x[::-1]), oz_dikkat(x)[::-1])
esit2 = np.allclose(oz_dikkat(x[::-1] + PE), oz_dikkat(x + PE)[::-1])
print(esit1, esit2)
Test yourself
-
Without positional encoding, why do "cat chased dog" and "dog chased cat" get confused?
- Softmax is faulty
- Scaling is missing
- Attention doesn't see order; the same words produce the same representations
- The vocabulary is too small
Show answer
Attention doesn't see order; the same words produce the same representations — If the word set is the same, without order information the representations are the same.
-
Place the sine into the even dimensions.
PE[:, 0::____] = np.sin(aci)
Show answer
PE[:, 0::2] = np.sin(aci)
-
Match.
- PE[0, 0::2]
- All 0 (sin 0)
- PE[0, 1::2]
- All 1 (cos 0)
- low dimension
- Fast oscillation
- RoPE
- Relative position
Next lesson: TransformerEncoderLayer
