Tokenization
- Machine Learning course
- Expert
- Lesson 19/44
- 5 min
BERT splits words into subword tokens.
Subword tokenization
Putting all words in the vocabulary is impossible (especially in agglutinative languages like Turkish). BERT uses WordPiece: a word is split into the longest pieces in the vocabulary; word-internal pieces are marked with ##:
"oyuncular" -> ["oyun", "##cu", "##lar"]
"güzeldi" -> ["güzel", "##di"]That way a vocabulary of ~30 000 pieces covers an unlimited number of words; if no piece is found, [UNK] is used.
Greedy longest match
The algorithm starts from the left, takes the longest prefix found in the vocabulary, then repeats the same on the rest with the ## prefix:
def wordpiece(kelime):
parca, bas = [], 0
while bas < len(kelime):
son = len(kelime)
while son > bas:
aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
if aday in sozluk:
break
son -= 1
if son == bas:
return ["[UNK]"]
parca.append(aday)
bas = son
return parcaIn Hugging Face, AutoTokenizer.from_pretrained(...) provides a fast version of this (written in Rust).
Practice WordPiece
Write the wordpiece(kelime) function with the given vocabulary. Split each word of the sentence "oyuncular film güzeldi masa", concatenate all the pieces into one list and print it. Then print the pieces' ids.
Starter code
sozluk = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "[MASK]": 4,
"film": 5, "çok": 6, "güzel": 7, "##di": 8, "oyun": 9, "##cu": 10,
"##lar": 11, "iyi": 12, "kötü": 13, "##ydi": 14}
metin = "oyuncular film güzeldi masa"
# buraya yazSample solution
sozluk = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "[MASK]": 4,
"film": 5, "çok": 6, "güzel": 7, "##di": 8, "oyun": 9, "##cu": 10,
"##lar": 11, "iyi": 12, "kötü": 13, "##ydi": 14}
metin = "oyuncular film güzeldi masa"
def wordpiece(kelime):
parca, bas = [], 0
while bas < len(kelime):
son = len(kelime)
while son > bas:
aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
if aday in sozluk:
break
son -= 1
if son == bas:
return ["[UNK]"]
parca.append(aday)
bas = son
return parca
parcalar = [p for k in metin.split() for p in wordpiece(k)]
print(parcalar)
print([sozluk[p] for p in parcalar])Expected output
['oyun', '##cu', '##lar', 'film', 'güzel', '##di', '[UNK]'] [9, 10, 11, 5, 7, 8, 1]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 In WordPiece, what does "##lar" mean?
- End of sentence
- A continuation piece, not the start of a word Correct answer
- A comment line
- An unknown word
Explanation: A ##-prefixed piece is attached to the previous piece.
2 Mark the word-internal piece.
aday = kelime[bas:son] if bas == 0 else "____" + kelime[bas:son]aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]Explanation: It is BERT's WordPiece prefix.
3 Match the tokenization.
| Term | Meaning |
|---|---|
oyuncular |
oyun ##cu ##lar |
güzeldi |
güzel ##di |
kötüydü |
kötü ##ydi |
masa |
[UNK] |