Masked LM
- Machine Learning course
- Expert
- Lesson 18/44
- 5 min
BERT masks random words during training and predicts them.
Masked language model
BERT's pre-training task is the masked language model (MLM): 15% of the tokens are selected and the model predicts them from context. The 80/10/10 rule is applied to the selected ones:
- 80% are replaced with
[MASK], - 10% are replaced with a random word,
- 10% are left as they are.
"film çok [MASK] di" -> the model must predict "güzel"The loss is computed only at the selected positions; no prediction is required for the other 85%.
Implementing the masking
Three groups are separated with random numbers. The label array is -100 at unselected positions; PyTorch's cross-entropy ignores this value:
secim = rng.random(n) < 0.15
r = rng.random(n)
maskeli = secim & (r < 0.8)
rastgele = secim & (r >= 0.8) & (r < 0.9)
giris = ids.copy()
giris[maskeli] = MASK_ID
giris[rastgele] = rng.integers(5, V, rastgele.sum())
etiket = np.where(secim, ids, -100)Special tokens ([CLS], [SEP], [PAD]) are excluded from masking.
Practice MLM masking
Apply masking with the 80/10/10 rule to a random sequence of 100 000 tokens. Print the selected ratio and, among the selected, the [MASK], random and unchanged ratios with round(..., 2) on one line. Then print with == that the number of labels that aren't -100 equals the number selected.
Starter code
import numpy as np
rng = np.random.default_rng(0)
V, MASK_ID, n = 15, 4, 100_000
ids = rng.integers(5, V, n)
# buraya yazSample solution
import numpy as np
rng = np.random.default_rng(0)
V, MASK_ID, n = 15, 4, 100_000
ids = rng.integers(5, V, n)
secim = rng.random(n) < 0.15
r = rng.random(n)
maskeli = secim & (r < 0.8)
rastgele = secim & (r >= 0.8) & (r < 0.9)
ayni = secim & (r >= 0.9)
giris = ids.copy()
giris[maskeli] = MASK_ID
giris[rastgele] = rng.integers(5, V, rastgele.sum())
etiket = np.where(secim, ids, -100)
k = secim.sum()
print(round(secim.mean(), 2), round(maskeli.sum() / k, 2), round(rastgele.sum() / k, 2), round(ayni.sum() / k, 2))
print((etiket != -100).sum() == k)Expected output
0.15 0.8 0.1 0.1 True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why are some of the selected tokens left as they are instead of [MASK]?
- To shrink the vocabulary
- To zero the loss
- To reduce the gap between training and use, since [MASK] never appears in fine-tuning Correct answer
- To speed up the computation
Explanation: The model is forced to learn representations of real words too.
2 Exclude unselected positions from the loss.
etiket = np.where(secim, ids, ____)etiket = np.where(secim, ids, -100)Explanation: CrossEntropyLoss's default ignore_index is -100.
3 Match the ratios.
| Term | Meaning |
|---|---|
selection |
15% |
[MASK] |
80% of the selected |
random word |
10% of the selected |
unchanged |
The remaining 10% of the selected |