Positional encoding
- Machine Learning course
- Expert
- Lesson 10/44
- 5 min
Positional encoding gives the model position information in the sequence.
Positional encoding
Attention is unaware of position: if you shuffle the order of the words, the outputs are shuffled the same way (permutation-equivariant). To convey order, a positional encoding is added to the input. The original paper uses sine/cosine:
def konum_kodu(n, d):
pos = np.arange(n)[:, None]
i = np.arange(0, d, 2)
aci = pos / 10000 ** (i / d)
PE = np.zeros((n, d))
PE[:, 0::2] = np.sin(aci)
PE[:, 1::2] = np.cos(aci)
return PE
x = gomme + konum_kodu(n, d)Low dimensions oscillate fast and high dimensions slowly; this way every position gets a unique "fingerprint".
Permutation equivariance
You can see this numerically: reversing the rows and applying self-attention is the same as reversing self-attention's output. Once the positional encoding is added, this equality breaks:
ters = x[::-1]
print(np.allclose(oz_dikkat(ters), oz_dikkat(x)[::-1])) # True
xp = x + konum_kodu(n, d)
tp = x[::-1] + konum_kodu(n, d)
print(np.allclose(oz_dikkat(tp), oz_dikkat(xp)[::-1])) # FalseModels like BERT and GPT use learned position embeddings instead of fixed sines; newer models prefer relative methods such as RoPE.
Practice Sinusoidal positional code
Write the konum_kodu(n, d) function. Print the shape and first row of konum_kodu(4, 6) on separate lines, then its second row with np.round(..., 3). Finally print on one line with np.allclose that, with the given oz_dikkat, reversing the rows is equivariant without the positional code and not equivariant with it.
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
def oz_dikkat(x):
return softmax(x @ x.T / np.sqrt(x.shape[1])) @ x
x = np.random.default_rng(0).normal(size=(4, 6))
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
def oz_dikkat(x):
return softmax(x @ x.T / np.sqrt(x.shape[1])) @ x
x = np.random.default_rng(0).normal(size=(4, 6))
def konum_kodu(n, d):
pos = np.arange(n)[:, None]
i = np.arange(0, d, 2)
aci = pos / 10000 ** (i / d)
PE = np.zeros((n, d))
PE[:, 0::2] = np.sin(aci)
PE[:, 1::2] = np.cos(aci)
return PE
PE = konum_kodu(4, 6)
print(PE.shape)
print(PE[0])
print(np.round(PE[1], 3))
esit1 = np.allclose(oz_dikkat(x[::-1]), oz_dikkat(x)[::-1])
esit2 = np.allclose(oz_dikkat(x[::-1] + PE), oz_dikkat(x + PE)[::-1])
print(esit1, esit2)Expected output
(4, 6) [0. 1. 0. 1. 0. 1.] [0.841 0.54 0.046 0.999 0.002 1. ] True False
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Without positional encoding, why do "cat chased dog" and "dog chased cat" get confused?
- Softmax is faulty
- Scaling is missing
- Attention doesn't see order; the same words produce the same representations Correct answer
- The vocabulary is too small
Explanation: If the word set is the same, without order information the representations are the same.
2 Place the sine into the even dimensions.
PE[:, 0::____] = np.sin(aci)PE[:, 0::2] = np.sin(aci)Explanation: 0::2 selects the even indices, 1::2 the odd ones.
3 Match.
| Term | Meaning |
|---|---|
PE[0, 0::2] |
All 0 (sin 0) |
PE[0, 1::2] |
All 1 (cos 0) |
low dimension |
Fast oscillation |
RoPE |
Relative position |