Feature extraction
- Machine Learning course
- Advanced
- Lesson 66/78
- 5 min
Feature extraction freezes the pre-trained model's feature extraction layers.
Feature extraction
The simplest transfer method: freeze the body completely and train only the new head. The body becomes a fixed feature extractor; training is fast and works even with very little data:
for p in govde.parameters():
p.requires_grad = False
model = nn.Sequential(govde, nn.Linear(32, 5))
opt = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)An even faster way: extract the features of all images once and save them, then train a simple model like logistic regression on those features.
Precomputing features
Because the body is frozen, recomputing the same features every epoch is needless. The features are extracted once and a small model is trained on them:
govde.eval()
with torch.no_grad():
F_train = govde(X_train) # (N, 32)
F_test = govde(X_test)
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(max_iter=1000).fit(F_train.numpy(), y_train.numpy())This method takes a few seconds even on thousands of images; its only drawback is that data augmentation can't be used.
Practice Freeze the body
Freeze govde and build nn.Sequential(govde, nn.Linear(32, 5)). Print the total and trainable parameter counts on one line. Backpropagate a loss on a random input (model(x).sum()); print on one line that the first convolution's weight gradient is None and the head's gradient isn't.
Starter code
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
x = torch.randn(2, 3, 16, 16)
# buraya yazSample solution
import torch
from torch import nn
torch.manual_seed(0)
govde = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)
x = torch.randn(2, 3, 16, 16)
for p in govde.parameters():
p.requires_grad = False
model = nn.Sequential(govde, nn.Linear(32, 5))
print(sum(p.numel() for p in model.parameters()), sum(p.numel() for p in model.parameters() if p.requires_grad))
model(x).sum().backward()
print(govde[0].weight.grad is None, model[1].weight.grad is not None)Expected output
5253 165 True True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 After backpropagation with a frozen body, what is the <code>.grad</code> of the body parameters?
- An error
- None Correct answer
- A zero tensor
- The normal gradient
Explanation: No gradient is computed for requires_grad=False parameters.
2 Give only the trainable parameters to the optimiser.
opt = torch.optim.Adam(____(lambda p: p.requires_grad, model.parameters()), lr=1e-3)opt = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)Explanation: filter removes the frozen parameters.
3 Match the methods.
| Term | Meaning |
|---|---|
feature extraction |
Body fully frozen |
fine-tuning |
Body trained too |
precomputing |
Features extracted once |
filter(requires_grad) |
Select the trainable ones |