Machine Learning
Fine-tuning
BERT is fine-tuned for tasks like classification.
Fine-tuning
In fine-tuning a small task layer (e.g. a 768 → 2 linear layer) is added on top of the pre-trained BERT and the whole model is trained for a few epochs with a small learning rate:
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
"dbmdz/bert-base-turkish-cased", num_labels=2)
# typical: lr = 2e-5, 2-4 epochs, batch 16-32, warmup + linear decay
With little data or limited hardware the body can be frozen and only the head trained; this is faster but usually gives lower accuracy.
Warmup and decay schedule
In BERT fine-tuning the learning rate first warms up linearly from zero to the target, then decays linearly toward zero. Warmup keeps the randomly initialized head from ruining the body in the first steps:
def lr(adim, toplam, isinma, tepe=2e-5):
if adim < isinma:
return tepe * adim / isinma
return tepe * (toplam - adim) / (toplam - isinma)
In Hugging Face, get_linear_schedule_with_warmup applies the same schedule.
Exercise Schedule and trainable parameters
Write the lr(adim, toplam, isinma) function. For toplam=1000, isinma=100, print the rates at steps 0, 50, 100, 550 and 1000 on one line. Then print on one line the number of trained parameters with full fine-tuning and with a frozen body when a 768 → 2 classification head is added on top of BERT-base (109 482 240 parameters).
Show example solution
govde = 109_482_240
def lr(adim, toplam, isinma, tepe=2e-5):
if adim < isinma:
return tepe * adim / isinma
return tepe * (toplam - adim) / (toplam - isinma)
print(*[lr(a, 1000, 100) for a in [0, 50, 100, 550, 1000]])
bas = 768 * 2 + 2
print(govde + bas, bas)
Test yourself
-
Why is the learning rate chosen small (e.g. 2e-5) in fine-tuning?
- Not to destroy pre-learned knowledge (to avoid catastrophic forgetting)
- To speed up training
- To save memory
- Because tokenization requires it
Show answer
Not to destroy pre-learned knowledge (to avoid catastrophic forgetting) — Large steps quickly ruin the weights learned in pre-training.
-
Increase the rate linearly during warmup.
return tepe * adim / ____
Show answer
return tepe * adim / isinma
-
Match.
- the whole model is trained
- Full fine-tuning
- head only
- Frozen body
- warm-up
- Rate rises from zero
- 2e-5
- Typical learning rate
Next lesson: CLS token
