Fine-tuning
- Machine Learning course
- Advanced
- Lesson 65/78
- 5 min
Fine-tuning retrains some layers of a pre-trained model.
Fine-tuning
Fine-tuning means starting from pretrained weights and training the whole network (or its upper layers) on the new data with a small learning rate. The small rate prevents the valuable general features from being destroyed:
opt = torch.optim.Adam([
{"params": govde.parameters(), "lr": 1e-4}, # body: small steps
{"params": yeni_bas.parameters(), "lr": 1e-3}, # new head: large steps
])Because the new head starts random it must learn faster; the body only makes fine corrections.
Gradual fine-tuning
A common strategy has two stages: first the body is frozen and only the head is trained; once the head is reasonable, the body is opened and trained along with it at a small rate:
# stage 1
for p in govde.parameters():
p.requires_grad = False
egit(bas, epoch=3)
# stage 2
for p in govde.parameters():
p.requires_grad = True
egit(tum_model, epoch=5, lr=1e-4)That way the head's random start is corrected before it can harm the body.
Practice Parameter groups
Build an Adam optimiser with lr=1e-4 for govde and lr=1e-3 for a new head nn.Linear(32, 5). Print the number of groups and each group's learning rate as a list on one line. Then print each group's parameter count as a list.
Starter code
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
# buraya yazSample solution
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
yeni_bas = nn.Linear(32, 5)
opt = torch.optim.Adam([{"params": govde.parameters(), "lr": 1e-4}, {"params": yeni_bas.parameters(), "lr": 1e-3}])
print(len(opt.param_groups), [g["lr"] for g in opt.param_groups])
print([sum(p.numel() for p in g["params"]) for g in opt.param_groups])Expected output
2 [0.0001, 0.001] [5088, 165]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What is the risk of giving the body a large learning rate in fine-tuning?
- There is no risk
- Memory overflow
- Destroying the pre-learned features Correct answer
- Training slowing down
Explanation: The random head's large gradients can wreck the body in the first steps.
2 Give the body a small learning rate.
{"params": govde.parameters(), "____": 1e-4}{"params": govde.parameters(), "lr": 1e-4}Explanation: The lr key is used in a parameter-group dict.
3 Match the settings.
| Term | Meaning |
|---|---|
body lr |
1e-4 (small) |
head lr |
1e-3 (large) |
stage 1 |
Body frozen |
stage 2 |
Whole model |