Machine Learning
CLS token
The CLS token carries a summary of the entire sequence for classification tasks.
[CLS] and [SEP]
[CLS] is placed at the start of BERT's input and [SEP] at the end of each sentence. In sentence-pair tasks (natural language inference, question answering) segment ids (token_type_ids) tell which token belongs to which sentence:
[CLS] film güzel ##di [SEP] oyun ##cu ##lar iyi [SEP]
0 0 0 0 0 1 1 1 1 1 <- token_type_ids
In classification the [CLS] vector of the last layer (position 0) is used as a summary of the whole sentence; thanks to attention this position gathers information from all tokens.
Attention mask and padding
The inputs in a batch are padded to the same length; attention_mask is 1 on real tokens and 0 on padding. The tokenizer returns the three arrays together:
{"input_ids": [2, 5, 7, 8, 3, 0, 0],
"token_type_ids": [0, 0, 0, 0, 0, 0, 0],
"attention_mask": [1, 1, 1, 1, 1, 0, 0]}
BERT's longest input is 512 tokens; longer texts are truncated or split into chunks.
Exercise Build a BERT input
Write the kodla(a, b=None, n=12) function with the given wordpiece and vocabulary: [CLS] a [SEP] (plus b [SEP] if given), segment ids (the second sentence is 1), padding and the attention mask. Print the three arrays of the result of kodla("film güzeldi", "oyuncular iyi") on separate lines.
Show example solution
sozluk = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "[MASK]": 4,
"film": 5, "çok": 6, "güzel": 7, "##di": 8, "oyun": 9, "##cu": 10,
"##lar": 11, "iyi": 12, "kötü": 13, "##ydi": 14}
def wordpiece(kelime):
parca, bas = [], 0
while bas < len(kelime):
son = len(kelime)
while son > bas:
aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
if aday in sozluk:
break
son -= 1
if son == bas:
return ["[UNK]"]
parca.append(aday)
bas = son
return parca
def parcala(metin):
return [p for k in metin.split() for p in wordpiece(k)]
def kodla(a, b=None, n=12):
parcalar = ["[CLS]"] + parcala(a) + ["[SEP]"]
tip = [0] * len(parcalar)
if b:
ek = parcala(b) + ["[SEP]"]
parcalar += ek
tip += [1] * len(ek)
ids = [sozluk[p] for p in parcalar]
maske = [1] * len(ids)
dolgu = n - len(ids)
return ids + [0] * dolgu, tip + [0] * dolgu, maske + [0] * dolgu
for dizi in kodla("film güzeldi", "oyuncular iyi"):
print(dizi)
Test yourself
-
Which position's vector does BERT's classification head use?
- The last [SEP]
- The sum of all positions
- A random position
- [CLS] (position 0)
Show answer
[CLS] (position 0) — The next-sentence task in pre-training was also done via [CLS].
-
End the sentence with the separator.
parcalar = ["[CLS]"] + a + ["____"]
Show answer
parcalar = ["[CLS]"] + a + ["[SEP]"]
-
Match the arrays.
- input_ids
- Token ids
- token_type_ids
- Which sentence
- attention_mask
- Real or padding
- [CLS] output
- Sentence summary
Next lesson: Summary - BERT
