Fine-tuning
- Machine Learning course
- Expert
- Lesson 20/44
- 5 min
BERT is fine-tuned for tasks like classification.
Fine-tuning
In fine-tuning a small task layer (e.g. a 768 → 2 linear layer) is added on top of the pre-trained BERT and the whole model is trained for a few epochs with a small learning rate:
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
"dbmdz/bert-base-turkish-cased", num_labels=2)
# typical: lr = 2e-5, 2-4 epochs, batch 16-32, warmup + linear decayWith little data or limited hardware the body can be frozen and only the head trained; this is faster but usually gives lower accuracy.
Warmup and decay schedule
In BERT fine-tuning the learning rate first warms up linearly from zero to the target, then decays linearly toward zero. Warmup keeps the randomly initialized head from ruining the body in the first steps:
def lr(adim, toplam, isinma, tepe=2e-5):
if adim < isinma:
return tepe * adim / isinma
return tepe * (toplam - adim) / (toplam - isinma)In Hugging Face, get_linear_schedule_with_warmup applies the same schedule.
Practice Schedule and trainable parameters
Write the lr(adim, toplam, isinma) function. For toplam=1000, isinma=100, print the rates at steps 0, 50, 100, 550 and 1000 on one line. Then print on one line the number of trained parameters with full fine-tuning and with a frozen body when a 768 → 2 classification head is added on top of BERT-base (109 482 240 parameters).
Starter code
govde = 109_482_240
# buraya yazSample solution
govde = 109_482_240
def lr(adim, toplam, isinma, tepe=2e-5):
if adim < isinma:
return tepe * adim / isinma
return tepe * (toplam - adim) / (toplam - isinma)
print(*[lr(a, 1000, 100) for a in [0, 50, 100, 550, 1000]])
bas = 768 * 2 + 2
print(govde + bas, bas)Expected output
0.0 1e-05 2e-05 1e-05 0.0 109483778 1538
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why is the learning rate chosen small (e.g. 2e-5) in fine-tuning?
- Not to destroy pre-learned knowledge (to avoid catastrophic forgetting) Correct answer
- To speed up training
- To save memory
- Because tokenization requires it
Explanation: Large steps quickly ruin the weights learned in pre-training.
2 Increase the rate linearly during warmup.
return tepe * adim / ____return tepe * adim / isinmaExplanation: When adim = isinma, the rate reaches the peak.
3 Match.
| Term | Meaning |
|---|---|
the whole model is trained |
Full fine-tuning |
head only |
Frozen body |
warm-up |
Rate rises from zero |
2e-5 |
Typical learning rate |