Summary - Transfer learning
- Machine Learning course
- Advanced
- Lesson 68/78
- 6 min
In this group you learned pre-trained models, fine-tuning, and feature extraction.
Summary: transfer learning
In this group:
- General features learned on large data are moved to new, small tasks.
- The body is kept and the head is replaced for the new class count (
in_features). - The pretrained model's preprocessing (size, normalisation) is applied as is.
- Feature extraction: body frozen; fine-tuning: small lr, parameter groups, gradual unfreezing.
- How much to train depends on the amount of data and the domain similarity.
Practice Precomputed features
Extract the features of 10 random images of size (3, 16, 16) with govde.eval() and torch.no_grad(). Print the feature matrix's shape and its requires_grad value on one line. Then convert the features to NumPy (.numpy()) and print the array's shape.
Starter code
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
X = torch.randn(10, 3, 16, 16)
# buraya yazSample solution
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
X = torch.randn(10, 3, 16, 16)
govde.eval()
with torch.no_grad():
F = govde(X)
print(F.shape, F.requires_grad)
print(F.numpy().shape)Expected output
torch.Size([10, 32]) False (10, 32)
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 With 200 labelled cat/dog photos, which is the best starting point?
- Training a large CNN from scratch
- Logistic regression (raw pixels)
- KMeans
- Freezing an ImageNet model's body and training a new head Correct answer
Explanation: Little data and a similar domain: feature extraction is the most efficient path.
2 Extract features in evaluation mode.
Question
govde.____()
with torch.no_grad():
F = govde(X)
Answer
govde.eval()
with torch.no_grad():
F = govde(X)Explanation: eval() puts BatchNorm and dropout into fixed behaviour.
3 Match the concepts.
| Term | Meaning |
|---|---|
backbone |
Body |
head |
Head |
freeze |
Freezing |
fine-tune |
Fine-tuning |