Summary - BERT
- Machine Learning course
- Expert
- Lesson 22/44
- 6 min
In this group you learned BERT, masked LM, and fine-tuning.
Summary: BERT
In this group:
- BERT is a stack of bidirectional transformer encoders (base: 12 layers, 768, ~110M).
- Pre-training: 15% of the tokens are selected and corrupted with the 80/10/10 rule; the loss is only on the selected ones.
- WordPiece splits words into the longest vocabulary pieces; continuation pieces are marked with
##. - Fine-tuning: small learning rate, warmup + linear decay, a small task head.
- Input:
[CLS] … [SEP] … [SEP], segment ids and the attention mask; classification uses the [CLS] vector.
Practice An end-to-end MLM example
With the given helpers, encode the sentence "film çok güzeldi" as [CLS] … [SEP]. Replace the piece "güzel" with [MASK] and build the label array (the real id only at that position, -100 elsewhere). Print the input and the labels on separate lines, then the tokens of the masked input (converted back from ids).
Starter code
sozluk = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "[MASK]": 4,
"film": 5, "çok": 6, "güzel": 7, "##di": 8, "oyun": 9, "##cu": 10,
"##lar": 11, "iyi": 12, "kötü": 13, "##ydi": 14}
def wordpiece(kelime):
parca, bas = [], 0
while bas < len(kelime):
son = len(kelime)
while son > bas:
aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
if aday in sozluk:
break
son -= 1
if son == bas:
return ["[UNK]"]
parca.append(aday)
bas = son
return parca
ters = {v: k for k, v in sozluk.items()}
# buraya yazSample solution
sozluk = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "[MASK]": 4,
"film": 5, "çok": 6, "güzel": 7, "##di": 8, "oyun": 9, "##cu": 10,
"##lar": 11, "iyi": 12, "kötü": 13, "##ydi": 14}
def wordpiece(kelime):
parca, bas = [], 0
while bas < len(kelime):
son = len(kelime)
while son > bas:
aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
if aday in sozluk:
break
son -= 1
if son == bas:
return ["[UNK]"]
parca.append(aday)
bas = son
return parca
ters = {v: k for k, v in sozluk.items()}
parcalar = ["[CLS]"] + [p for k in "film çok güzeldi".split() for p in wordpiece(k)] + ["[SEP]"]
ids = [sozluk[p] for p in parcalar]
i = parcalar.index("güzel")
giris = ids.copy()
giris[i] = sozluk["[MASK]"]
etiket = [ids[j] if j == i else -100 for j in range(len(ids))]
print(giris)
print(etiket)
print([ters[t] for t in giris])Expected output
[2, 5, 6, 4, 8, 3] [-100, -100, -100, 7, -100, -100] ['[CLS]', 'film', 'çok', '[MASK]', '##di', '[SEP]']
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 In MLM, about how many tokens of a 1000-token text are replaced with [MASK]?
- 800
- 15
- 120 Correct answer
- 150
Explanation: 1000 × 0.15 = 150 are selected; 80% of them = 120.
2 Give the label to be ignored in the loss.
Question
kayip = nn.CrossEntropyLoss(ignore_index=____)
Answer
kayip = nn.CrossEntropyLoss(ignore_index=-100)Explanation: Unselected positions have the label -100.
3 Match.
| Term | Meaning |
|---|---|
MLM |
Pre-training task |
WordPiece |
Subword splitting |
[CLS] |
Classification vector |
warm-up |
Stable start |