Summary - Transfer learning

In this group you learned pre-trained models, fine-tuning, and feature extraction.

Summary: transfer learning

In this group:

General features learned on large data are moved to new, small tasks.
The body is kept and the head is replaced for the new class count (in_features).
The pretrained model's preprocessing (size, normalisation) is applied as is.
Feature extraction: body frozen; fine-tuning: small lr, parameter groups, gradual unfreezing.
How much to train depends on the amount of data and the domain similarity.

Exercise Precomputed features

Extract the features of 10 random images of size (3, 16, 16) with govde.eval() and torch.no_grad(). Print the feature matrix's shape and its requires_grad value on one line. Then convert the features to NumPy (.numpy()) and print the array's shape.

Show example solution

import torch
from torch import nn

torch.manual_seed(0)
govde = nn.Sequential(
    nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
    nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)

X = torch.randn(10, 3, 16, 16)

govde.eval()
with torch.no_grad():
    F = govde(X)
print(F.shape, F.requires_grad)
print(F.numpy().shape)

Test yourself

With 200 labelled cat/dog photos, which is the best starting point?
- Training a large CNN from scratch
- Logistic regression (raw pixels)
- KMeans
- Freezing an ImageNet model's body and training a new head
Show answer

Freezing an ImageNet model's body and training a new head — Little data and a similar domain: feature extraction is the most efficient path.

Extract features in evaluation mode.

govde.____()
with torch.no_grad():
    F = govde(X)

Show answer

govde.eval()
with torch.no_grad():
    F = govde(X)

Match the concepts.

backbone

Body
head

Head
freeze

Freezing
fine-tune

Fine-tuning

Next lesson: What is an optimizer?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies