Machine Learning
Feature extraction
Feature extraction freezes the pre-trained model's feature extraction layers.
Feature extraction
The simplest transfer method: freeze the body completely and train only the new head. The body becomes a fixed feature extractor; training is fast and works even with very little data:
for p in govde.parameters():
p.requires_grad = False
model = nn.Sequential(govde, nn.Linear(32, 5))
opt = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)
An even faster way: extract the features of all images once and save them, then train a simple model like logistic regression on those features.
Precomputing features
Because the body is frozen, recomputing the same features every epoch is needless. The features are extracted once and a small model is trained on them:
govde.eval()
with torch.no_grad():
F_train = govde(X_train) # (N, 32)
F_test = govde(X_test)
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(max_iter=1000).fit(F_train.numpy(), y_train.numpy())
This method takes a few seconds even on thousands of images; its only drawback is that data augmentation can't be used.
Exercise Freeze the body
Freeze govde and build nn.Sequential(govde, nn.Linear(32, 5)). Print the total and trainable parameter counts on one line. Backpropagate a loss on a random input (model(x).sum()); print on one line that the first convolution's weight gradient is None and the head's gradient isn't.
Show example solution
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
x = torch.randn(2, 3, 16, 16)
for p in govde.parameters():
p.requires_grad = False
model = nn.Sequential(govde, nn.Linear(32, 5))
print(sum(p.numel() for p in model.parameters()), sum(p.numel() for p in model.parameters() if p.requires_grad))
model(x).sum().backward()
print(govde[0].weight.grad is None, model[1].weight.grad is not None)
Test yourself
-
After backpropagation with a frozen body, what is the <code>.grad</code> of the body parameters?
- An error
- None
- A zero tensor
- The normal gradient
Show answer
None — No gradient is computed for requires_grad=False parameters.
-
Give only the trainable parameters to the optimiser.
opt = torch.optim.Adam(____(lambda p: p.requires_grad, model.parameters()), lr=1e-3)
Show answer
opt = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)
-
Match the methods.
- feature extraction
- Body fully frozen
- fine-tuning
- Body trained too
- precomputing
- Features extracted once
- filter(requires_grad)
- Select the trainable ones
Next lesson: Advantages
