Summary - BERT

In this group you learned BERT, masked LM, and fine-tuning.

Summary: BERT

In this group:

BERT is a stack of bidirectional transformer encoders (base: 12 layers, 768, ~110M).
Pre-training: 15% of the tokens are selected and corrupted with the 80/10/10 rule; the loss is only on the selected ones.
WordPiece splits words into the longest vocabulary pieces; continuation pieces are marked with ##.
Fine-tuning: small learning rate, warmup + linear decay, a small task head.
Input: [CLS] … [SEP] … [SEP], segment ids and the attention mask; classification uses the [CLS] vector.

Exercise An end-to-end MLM example

With the given helpers, encode the sentence "film çok güzeldi" as [CLS] … [SEP]. Replace the piece "güzel" with [MASK] and build the label array (the real id only at that position, -100 elsewhere). Print the input and the labels on separate lines, then the tokens of the masked input (converted back from ids).

Show example solution

sozluk = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "[MASK]": 4,
          "film": 5, "çok": 6, "güzel": 7, "##di": 8, "oyun": 9, "##cu": 10,
          "##lar": 11, "iyi": 12, "kötü": 13, "##ydi": 14}

def wordpiece(kelime):
    parca, bas = [], 0
    while bas < len(kelime):
        son = len(kelime)
        while son > bas:
            aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
            if aday in sozluk:
                break
            son -= 1
        if son == bas:
            return ["[UNK]"]
        parca.append(aday)
        bas = son
    return parca

ters = {v: k for k, v in sozluk.items()}

parcalar = ["[CLS]"] + [p for k in "film çok güzeldi".split() for p in wordpiece(k)] + ["[SEP]"]
ids = [sozluk[p] for p in parcalar]
i = parcalar.index("güzel")
giris = ids.copy()
giris[i] = sozluk["[MASK]"]
etiket = [ids[j] if j == i else -100 for j in range(len(ids))]
print(giris)
print(etiket)
print([ters[t] for t in giris])

Test yourself

In MLM, about how many tokens of a 1000-token text are replaced with [MASK]?
- 800
- 15
- 120
- 150
Show answer

120 — 1000 × 0.15 = 150 are selected; 80% of them = 120.

Give the label to be ignored in the loss.

kayip = nn.CrossEntropyLoss(ignore_index=____)

Show answer

kayip = nn.CrossEntropyLoss(ignore_index=-100)

Match.

MLM

Pre-training task
WordPiece

Subword splitting
[CLS]

Classification vector
warm-up

Stable start

Next lesson: What is GPT?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies