CLS token
- Machine Learning course
- Expert
- Lesson 21/44
- 5 min
The CLS token carries a summary of the entire sequence for classification tasks.
[CLS] and [SEP]
[CLS] is placed at the start of BERT's input and [SEP] at the end of each sentence. In sentence-pair tasks (natural language inference, question answering) segment ids (token_type_ids) tell which token belongs to which sentence:
[CLS] film güzel ##di [SEP] oyun ##cu ##lar iyi [SEP]
0 0 0 0 0 1 1 1 1 1 <- token_type_idsIn classification the [CLS] vector of the last layer (position 0) is used as a summary of the whole sentence; thanks to attention this position gathers information from all tokens.
Attention mask and padding
The inputs in a batch are padded to the same length; attention_mask is 1 on real tokens and 0 on padding. The tokenizer returns the three arrays together:
{"input_ids": [2, 5, 7, 8, 3, 0, 0],
"token_type_ids": [0, 0, 0, 0, 0, 0, 0],
"attention_mask": [1, 1, 1, 1, 1, 0, 0]}BERT's longest input is 512 tokens; longer texts are truncated or split into chunks.
Practice Build a BERT input
Write the kodla(a, b=None, n=12) function with the given wordpiece and vocabulary: [CLS] a [SEP] (plus b [SEP] if given), segment ids (the second sentence is 1), padding and the attention mask. Print the three arrays of the result of kodla("film güzeldi", "oyuncular iyi") on separate lines.
Starter code
sozluk = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "[MASK]": 4,
"film": 5, "çok": 6, "güzel": 7, "##di": 8, "oyun": 9, "##cu": 10,
"##lar": 11, "iyi": 12, "kötü": 13, "##ydi": 14}
def wordpiece(kelime):
parca, bas = [], 0
while bas < len(kelime):
son = len(kelime)
while son > bas:
aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
if aday in sozluk:
break
son -= 1
if son == bas:
return ["[UNK]"]
parca.append(aday)
bas = son
return parca
# buraya yazSample solution
sozluk = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "[MASK]": 4,
"film": 5, "çok": 6, "güzel": 7, "##di": 8, "oyun": 9, "##cu": 10,
"##lar": 11, "iyi": 12, "kötü": 13, "##ydi": 14}
def wordpiece(kelime):
parca, bas = [], 0
while bas < len(kelime):
son = len(kelime)
while son > bas:
aday = kelime[bas:son] if bas == 0 else "##" + kelime[bas:son]
if aday in sozluk:
break
son -= 1
if son == bas:
return ["[UNK]"]
parca.append(aday)
bas = son
return parca
def parcala(metin):
return [p for k in metin.split() for p in wordpiece(k)]
def kodla(a, b=None, n=12):
parcalar = ["[CLS]"] + parcala(a) + ["[SEP]"]
tip = [0] * len(parcalar)
if b:
ek = parcala(b) + ["[SEP]"]
parcalar += ek
tip += [1] * len(ek)
ids = [sozluk[p] for p in parcalar]
maske = [1] * len(ids)
dolgu = n - len(ids)
return ids + [0] * dolgu, tip + [0] * dolgu, maske + [0] * dolgu
for dizi in kodla("film güzeldi", "oyuncular iyi"):
print(dizi)Expected output
[2, 5, 7, 8, 3, 9, 10, 11, 12, 3, 0, 0] [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Which position's vector does BERT's classification head use?
- The last [SEP]
- The sum of all positions
- A random position
- [CLS] (position 0) Correct answer
Explanation: The next-sentence task in pre-training was also done via [CLS].
2 End the sentence with the separator.
parcalar = ["[CLS]"] + a + ["____"]parcalar = ["[CLS]"] + a + ["[SEP]"]Explanation: Each sentence ends with [SEP].
3 Match the arrays.
| Term | Meaning |
|---|---|
input_ids |
Token ids |
token_type_ids |
Which sentence |
attention_mask |
Real or padding |
[CLS] output |
Sentence summary |