Advantages
- Machine Learning course
- Advanced
- Lesson 67/78
- 5 min
Transfer learning provides high performance with little data.
Advantages and limits
The gains of transfer learning: less data, shorter training, higher accuracy and more stable results. Its limits:
- Domain gap: if the pretraining data is very different from the new data (natural photos → X-rays), the gain shrinks.
- Input fit: channel count, size and normalisation must match what the model expects.
- Size: large models are slow on small devices.
With a large domain gap, more layers need fine-tuning.
How much to train?
A rough guide by data amount and domain similarity:
- Little data, similar domain: freeze the body, train only the head.
- Little data, different domain: also fine-tune the upper blocks.
- Lots of data, similar domain: fine-tune the whole network at a small rate.
- Lots of data, different domain: fine-tune everything or try training from scratch.
# open only the last block in a ResNet:
for ad, p in model.named_parameters():
p.requires_grad = ad.startswith("layer4") or ad.startswith("fc")Practice Selective unfreezing
First freeze all parameters. Then unfreeze the parameters whose names start with "0.3." (the body's second convolution) and the head layer ("1."). Print the trainable parameter names as a list, then their count. Finally turn a (2, 1, 16, 16) grey image into three channels with repeat and print its shape.
Starter code
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
model = nn.Sequential(govde, nn.Linear(32, 5))
gri = torch.zeros(2, 1, 16, 16)
# buraya yazSample solution
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
model = nn.Sequential(govde, nn.Linear(32, 5))
gri = torch.zeros(2, 1, 16, 16)
for ad, p in model.named_parameters():
p.requires_grad = ad.startswith("0.3.") or ad.startswith("1.")
print([ad for ad, p in model.named_parameters() if p.requires_grad])
print(sum(p.numel() for p in model.parameters() if p.requires_grad))
print(gri.repeat(1, 3, 1, 1).shape)Expected output
['0.3.weight', '0.3.bias', '1.weight', '1.bias'] 4805 torch.Size([2, 3, 16, 16])
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 How is a 3-channel ImageNet model used with greyscale (single-channel) medical images?
- By copying the channel three times or adapting the first layer Correct answer
- It can't be used
- By colourising the image
- No change is needed
Explanation: x.repeat(1, 3, 1, 1) replicates the single channel into three.
2 Replicate the single channel into three.
x3 = x.____(1, 3, 1, 1)x3 = x.repeat(1, 3, 1, 1)Explanation: repeat takes a repeat count for each axis.
3 Match the situations with the strategies.
| Term | Meaning |
|---|---|
Little data, similar domain |
Head only |
Little data, different domain |
Upper blocks + head |
Lots of data, similar domain |
Whole network, small lr |
Single-channel input |
repeat(1, 3, 1, 1) |