Machine Learning
Masked LM
BERT masks random words during training and predicts them.
Masked language model
BERT's pre-training task is the masked language model (MLM): 15% of the tokens are selected and the model predicts them from context. The 80/10/10 rule is applied to the selected ones:
- 80% are replaced with
[MASK], - 10% are replaced with a random word,
- 10% are left as they are.
"film çok [MASK] di" -> the model must predict "güzel"
The loss is computed only at the selected positions; no prediction is required for the other 85%.
Implementing the masking
Three groups are separated with random numbers. The label array is -100 at unselected positions; PyTorch's cross-entropy ignores this value:
secim = rng.random(n) < 0.15
r = rng.random(n)
maskeli = secim & (r < 0.8)
rastgele = secim & (r >= 0.8) & (r < 0.9)
giris = ids.copy()
giris[maskeli] = MASK_ID
giris[rastgele] = rng.integers(5, V, rastgele.sum())
etiket = np.where(secim, ids, -100)
Special tokens ([CLS], [SEP], [PAD]) are excluded from masking.
Exercise MLM masking
Apply masking with the 80/10/10 rule to a random sequence of 100 000 tokens. Print the selected ratio and, among the selected, the [MASK], random and unchanged ratios with round(..., 2) on one line. Then print with == that the number of labels that aren't -100 equals the number selected.
Show example solution
import numpy as np
rng = np.random.default_rng(0)
V, MASK_ID, n = 15, 4, 100_000
ids = rng.integers(5, V, n)
secim = rng.random(n) < 0.15
r = rng.random(n)
maskeli = secim & (r < 0.8)
rastgele = secim & (r >= 0.8) & (r < 0.9)
ayni = secim & (r >= 0.9)
giris = ids.copy()
giris[maskeli] = MASK_ID
giris[rastgele] = rng.integers(5, V, rastgele.sum())
etiket = np.where(secim, ids, -100)
k = secim.sum()
print(round(secim.mean(), 2), round(maskeli.sum() / k, 2), round(rastgele.sum() / k, 2), round(ayni.sum() / k, 2))
print((etiket != -100).sum() == k)
Test yourself
-
Why are some of the selected tokens left as they are instead of [MASK]?
- To shrink the vocabulary
- To zero the loss
- To reduce the gap between training and use, since [MASK] never appears in fine-tuning
- To speed up the computation
Show answer
To reduce the gap between training and use, since [MASK] never appears in fine-tuning — The model is forced to learn representations of real words too.
-
Exclude unselected positions from the loss.
etiket = np.where(secim, ids, ____)
Show answer
etiket = np.where(secim, ids, -100)
-
Match the ratios.
- selection
- 15%
- [MASK]
- 80% of the selected
- random word
- 10% of the selected
- unchanged
- The remaining 10% of the selected
Next lesson: Tokenization
