Machine Learning
What is transfer learning?
Transfer learning adapts a pre-trained model to a new task.
What is transfer learning?
Transfer learning moves what a model learned on a large dataset (like ImageNet's 1.2 million images) to a new and usually small problem. Because the lower layers learn general features like edges, textures and shapes, this knowledge helps in other tasks too.
import torchvision
model = torchvision.models.resnet18(weights="IMAGENET1K_V1") # pretrained
model.fc = nn.Linear(model.fc.in_features, 5) # a new 5-class task
While a CNN trained from scratch on a few hundred images stays weak, starting from a pretrained model often gives high accuracy.
Body and head
A classification network has two parts: the body (backbone), which extracts features, and the head, which produces class scores from them. In transfer learning the body is kept and the head is replaced according to the new task's class count:
# pretrained: govde -> Linear(512, 1000) (ImageNet's 1000 classes)
# new task: govde -> Linear(512, 5) (a new head, starts random)
The body's last layer is class-specific while the lower layers are more general; how much to retrain is chosen by the amount of data.
Exercise Replace the head
onceden (body + a 1000-class head), which imitates a "pretrained" model, is given. Build a new model: the same govde and nn.Linear(bas.in_features, 5). Print the old and new output shapes for a (4, 3, 32, 32) input on separate lines.
Show example solution
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
x = torch.zeros(4, 3, 32, 32)
yeni = nn.Sequential(govde, nn.Linear(bas.in_features, 5))
print(onceden(x).shape)
print(yeni(x).shape)
Test yourself
-
In which situation does transfer learning make the most difference?
- When the new task has little labelled data
- When there are millions of labelled examples
- When the data is tabular
- Never
Show answer
When the new task has little labelled data — Pre-learned general features provide knowledge that couldn't be learned from little data.
-
Replace the head layer for the new task.
model.fc = nn.Linear(model.fc.____, 5)
Show answer
model.fc = nn.Linear(model.fc.in_features, 5)
-
Match the concepts.
- body
- General features
- head
- Task-specific classifier
- pretrained
- Learned on large data
- in_features
- Body output size
Next lesson: Pre-trained model
