Machine Learning
Fine-tuning
Fine-tuning retrains some layers of a pre-trained model.
Fine-tuning
Fine-tuning means starting from pretrained weights and training the whole network (or its upper layers) on the new data with a small learning rate. The small rate prevents the valuable general features from being destroyed:
opt = torch.optim.Adam([
{"params": govde.parameters(), "lr": 1e-4}, # body: small steps
{"params": yeni_bas.parameters(), "lr": 1e-3}, # new head: large steps
])
Because the new head starts random it must learn faster; the body only makes fine corrections.
Gradual fine-tuning
A common strategy has two stages: first the body is frozen and only the head is trained; once the head is reasonable, the body is opened and trained along with it at a small rate:
# stage 1
for p in govde.parameters():
p.requires_grad = False
egit(bas, epoch=3)
# stage 2
for p in govde.parameters():
p.requires_grad = True
egit(tum_model, epoch=5, lr=1e-4)
That way the head's random start is corrected before it can harm the body.
Exercise Parameter groups
Build an Adam optimiser with lr=1e-4 for govde and lr=1e-3 for a new head nn.Linear(32, 5). Print the number of groups and each group's learning rate as a list on one line. Then print each group's parameter count as a list.
Show example solution
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
yeni_bas = nn.Linear(32, 5)
opt = torch.optim.Adam([{"params": govde.parameters(), "lr": 1e-4}, {"params": yeni_bas.parameters(), "lr": 1e-3}])
print(len(opt.param_groups), [g["lr"] for g in opt.param_groups])
print([sum(p.numel() for p in g["params"]) for g in opt.param_groups])
Test yourself
-
What is the risk of giving the body a large learning rate in fine-tuning?
- There is no risk
- Memory overflow
- Destroying the pre-learned features
- Training slowing down
Show answer
Destroying the pre-learned features — The random head's large gradients can wreck the body in the first steps.
-
Give the body a small learning rate.
{"params": govde.parameters(), "____": 1e-4}
Show answer
{"params": govde.parameters(), "lr": 1e-4}
-
Match the settings.
- body lr
- 1e-4 (small)
- head lr
- 1e-3 (large)
- stage 1
- Body frozen
- stage 2
- Whole model
Next lesson: Feature extraction
