Machine Learning
Advantages
Transfer learning provides high performance with little data.
Advantages and limits
The gains of transfer learning: less data, shorter training, higher accuracy and more stable results. Its limits:
- Domain gap: if the pretraining data is very different from the new data (natural photos → X-rays), the gain shrinks.
- Input fit: channel count, size and normalisation must match what the model expects.
- Size: large models are slow on small devices.
With a large domain gap, more layers need fine-tuning.
How much to train?
A rough guide by data amount and domain similarity:
- Little data, similar domain: freeze the body, train only the head.
- Little data, different domain: also fine-tune the upper blocks.
- Lots of data, similar domain: fine-tune the whole network at a small rate.
- Lots of data, different domain: fine-tune everything or try training from scratch.
# open only the last block in a ResNet:
for ad, p in model.named_parameters():
p.requires_grad = ad.startswith("layer4") or ad.startswith("fc")
Exercise Selective unfreezing
First freeze all parameters. Then unfreeze the parameters whose names start with "0.3." (the body's second convolution) and the head layer ("1."). Print the trainable parameter names as a list, then their count. Finally turn a (2, 1, 16, 16) grey image into three channels with repeat and print its shape.
Show example solution
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
model = nn.Sequential(govde, nn.Linear(32, 5))
gri = torch.zeros(2, 1, 16, 16)
for ad, p in model.named_parameters():
p.requires_grad = ad.startswith("0.3.") or ad.startswith("1.")
print([ad for ad, p in model.named_parameters() if p.requires_grad])
print(sum(p.numel() for p in model.parameters() if p.requires_grad))
print(gri.repeat(1, 3, 1, 1).shape)
Test yourself
-
How is a 3-channel ImageNet model used with greyscale (single-channel) medical images?
- By copying the channel three times or adapting the first layer
- It can't be used
- By colourising the image
- No change is needed
Show answer
By copying the channel three times or adapting the first layer — x.repeat(1, 3, 1, 1) replicates the single channel into three.
-
Replicate the single channel into three.
x3 = x.____(1, 3, 1, 1)
Show answer
x3 = x.repeat(1, 3, 1, 1)
-
Match the situations with the strategies.
- Little data, similar domain
- Head only
- Little data, different domain
- Upper blocks + head
- Lots of data, similar domain
- Whole network, small lr
- Single-channel input
- repeat(1, 3, 1, 1)
Next lesson: Summary - Transfer learning
