Machine Learning
What is BERT?
BERT is a bidirectional language understanding model based on transformers.
What is BERT?
BERT (Bidirectional Encoder Representations from Transformers, 2018) is a model made only of transformer encoder blocks, pre-trained on large amounts of text. Each word sees the context both to its left and to its right (bidirectional).
- Pre-training: a masked language model on unlabelled text.
- Fine-tuning: classification, question answering, entity recognition with little labelled data.
# BERT-base: 12 layers, d_model = 768, 12 heads, FFN 3072, ~110M parameters
# BERT-large: 24 layers, d_model = 1024, 16 heads, ~340M parameters
Unlike GPT, BERT is designed not to generate text but to understand it (to produce representations).
Where do 110 million come from?
Counting the parameters piece by piece shows where the model is "heavy":
V, L, T, d, f = 30522, 512, 2, 768, 3072
gomme = (V + L + T) * d + 2 * d # word + position + segment + LayerNorm
katman = 4 * (d * d + d) + 2 * d + (d * f + f + f * d + d) + 2 * d
pooler = d * d + d
print(gomme + 12 * katman + pooler)
The word embedding table alone is ~23 million; two thirds of each layer is in the FFN.
Exercise BERT-base parameters
Compute the parameter counts of the embeddings, a single layer and the pooler with the given settings. Print the three on one line, then the total (with 12 layers). Finally print the FFN's share of a layer with round(..., 3).
Show example solution
V, L, T, d, f = 30522, 512, 2, 768, 3072
gomme = (V + L + T) * d + 2 * d
ffn = d * f + f + f * d + d
katman = 4 * (d * d + d) + 2 * d + ffn + 2 * d
pooler = d * d + d
print(gomme, katman, pooler)
print(gomme + 12 * katman + pooler)
print(round(ffn / katman, 3))
Test yourself
-
What is the main architectural feature separating BERT from GPT?
- More layers
- Using an RNN
- Not using attention
- A bidirectional encoder; no causal mask
Show answer
A bidirectional encoder; no causal mask — BERT sees the whole sentence at each position; GPT sees only the left.
-
Count attention's four projections (Q, K, V, output).
dikkat = ____ * (d * d + d)
Show answer
dikkat = 4 * (d * d + d)
-
Match the BERT-base settings.
- layer
- 12
- d_model
- 768
- FFN size
- 3072
- vocabulary
- 30522
Next lesson: Masked LM
