CodeBlanks Logo Icon CodeBlanks
Login Register

What is BERT?

BERT is a bidirectional language understanding model based on transformers.

What is BERT?

BERT (Bidirectional Encoder Representations from Transformers, 2018) is a model made only of transformer encoder blocks, pre-trained on large amounts of text. Each word sees the context both to its left and to its right (bidirectional).

  • Pre-training: a masked language model on unlabelled text.
  • Fine-tuning: classification, question answering, entity recognition with little labelled data.
# BERT-base: 12 layers, d_model = 768, 12 heads, FFN 3072, ~110M parameters
# BERT-large: 24 layers, d_model = 1024, 16 heads, ~340M parameters

Unlike GPT, BERT is designed not to generate text but to understand it (to produce representations).

Where do 110 million come from?

Counting the parameters piece by piece shows where the model is "heavy":

V, L, T, d, f = 30522, 512, 2, 768, 3072
gomme = (V + L + T) * d + 2 * d         # word + position + segment + LayerNorm
katman = 4 * (d * d + d) + 2 * d + (d * f + f + f * d + d) + 2 * d
pooler = d * d + d
print(gomme + 12 * katman + pooler)

The word embedding table alone is ~23 million; two thirds of each layer is in the FFN.

Practice BERT-base parameters

Compute the parameter counts of the embeddings, a single layer and the pooler with the given settings. Print the three on one line, then the total (with 12 layers). Finally print the FFN's share of a layer with round(..., 3).

Starter code

model.py
V, L, T, d, f = 30522, 512, 2, 768, 3072

# buraya yaz

Sample solution

model.py
V, L, T, d, f = 30522, 512, 2, 768, 3072

gomme = (V + L + T) * d + 2 * d
ffn = d * f + f + f * d + d
katman = 4 * (d * d + d) + 2 * d + ffn + 2 * d
pooler = d * d + d
print(gomme, katman, pooler)
print(gomme + 12 * katman + pooler)
print(round(ffn / katman, 3))

Expected output

23837184 7087872 590592
109482240
0.666

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is the main architectural feature separating BERT from GPT?

  • More layers
  • Using an RNN
  • Not using attention
  • A bidirectional encoder; no causal mask Correct answer

Explanation: BERT sees the whole sentence at each position; GPT sees only the left.

2 Count attention's four projections (Q, K, V, output).

Question 
dikkat = ____ * (d * d + d)
Answer 
dikkat = 4 * (d * d + d)

Explanation: Q, K, V and the output projection: four d×d matrices and their biases.

3 Match the BERT-base settings.

Term Meaning
layer 12
d_model 768
FFN size 3072
vocabulary 30522

All lessons in Expert

1 Attention

  1. What is attention?
  2. Self-attention
  3. Query, Key, Value
  4. Scaled dot-product
  5. Multi-head
  6. Summary - Attention

2 Transformer

  1. What is a transformer?
  2. Encoder
  3. Decoder
  4. Positional encoding
  5. TransformerEncoderLayer
  6. Summary - Transformer

3 Project: Transformer

  1. Preparing data
  2. Attention layer
  3. Encoder block
  4. Results

4 BERT

  1. What is BERT?
  2. Masked LM
  3. Tokenization
  4. Fine-tuning
  5. CLS token
  6. Summary - BERT

5 GPT

  1. What is GPT?
  2. Autoregressive
  3. Decoder-only
  4. Causal mask
  5. Text generation
  6. Summary - GPT

6 GAN and VAE

  1. What is GAN?
  2. Generator
  3. Discriminator
  4. What is VAE?
  5. Encoder-Decoder
  6. Summary - GAN and VAE

7 Reinforcement learning

  1. What is RL?
  2. Agent
  3. Policy
  4. Reward
  5. Q-learning
  6. Summary - RL

8 Expert final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4