Machine Learning
Tokenization
BERT splits words into subword tokens.
Subword tokenization
Putting all words in the vocabulary is impossible (especially in agglutinative languages like Turkish). BERT uses WordPiece: a word is split into the longest pieces in the vocabulary; word-internal pieces are marked with ##:
"oyuncular" -> ["oyun", "##cu", "##lar"]
"güzeldi" -> ["güzel", "##di"]
That way a vocabulary of ~30 000 pieces covers an unlimited number of words; if no piece is found, [UNK] is used.
Greedy longest match
The algorithm starts from the left, takes the longest prefix found in the vocabulary, then repeats the same on the rest with the ## prefix:
def wordpiece(kelime):
parca, bas = [], 0
while bas < len(kelime):
son = len(kelime)
while son > bas:
aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
if aday in sozluk:
break
son -= 1
if son == bas:
return ["[UNK]"]
parca.append(aday)
bas = son
return parca
In Hugging Face, AutoTokenizer.from_pretrained(...) provides a fast version of this (written in Rust).
Exercise WordPiece
Write the wordpiece(kelime) function with the given vocabulary. Split each word of the sentence "oyuncular film güzeldi masa", concatenate all the pieces into one list and print it. Then print the pieces' ids.
Show example solution
sozluk = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "[MASK]": 4,
"film": 5, "çok": 6, "güzel": 7, "##di": 8, "oyun": 9, "##cu": 10,
"##lar": 11, "iyi": 12, "kötü": 13, "##ydi": 14}
metin = "oyuncular film güzeldi masa"
def wordpiece(kelime):
parca, bas = [], 0
while bas < len(kelime):
son = len(kelime)
while son > bas:
aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
if aday in sozluk:
break
son -= 1
if son == bas:
return ["[UNK]"]
parca.append(aday)
bas = son
return parca
parcalar = [p for k in metin.split() for p in wordpiece(k)]
print(parcalar)
print([sozluk[p] for p in parcalar])
Test yourself
-
In WordPiece, what does "##lar" mean?
- End of sentence
- A continuation piece, not the start of a word
- A comment line
- An unknown word
Show answer
A continuation piece, not the start of a word — A ##-prefixed piece is attached to the previous piece.
-
Mark the word-internal piece.
aday = kelime[bas:son] if bas == 0 else "____" + kelime[bas:son]
Show answer
aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
-
Match the tokenization.
- oyuncular
- oyun ##cu ##lar
- güzeldi
- güzel ##di
- kötüydü
- kötü ##ydi
- masa
- [UNK]
Next lesson: Fine-tuning
