CodeBlanks Logo Icon CodeBlanks
Login Register

What is transfer learning?

Transfer learning adapts a pre-trained model to a new task.

What is transfer learning?

Transfer learning moves what a model learned on a large dataset (like ImageNet's 1.2 million images) to a new and usually small problem. Because the lower layers learn general features like edges, textures and shapes, this knowledge helps in other tasks too.

import torchvision
model = torchvision.models.resnet18(weights="IMAGENET1K_V1")   # pretrained
model.fc = nn.Linear(model.fc.in_features, 5)                  # a new 5-class task

While a CNN trained from scratch on a few hundred images stays weak, starting from a pretrained model often gives high accuracy.

Body and head

A classification network has two parts: the body (backbone), which extracts features, and the head, which produces class scores from them. In transfer learning the body is kept and the head is replaced according to the new task's class count:

# pretrained: govde -> Linear(512, 1000)  (ImageNet's 1000 classes)
# new task:   govde -> Linear(512, 5)     (a new head, starts random)

The body's last layer is class-specific while the lower layers are more general; how much to retrain is chosen by the amount of data.

Practice Replace the head

onceden (body + a 1000-class head), which imitates a "pretrained" model, is given. Build a new model: the same govde and nn.Linear(bas.in_features, 5). Print the old and new output shapes for a (4, 3, 32, 32) input on separate lines.

Starter code

model.py
import torch
from torch import nn

torch.manual_seed(0)
govde = nn.Sequential(
    nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
    nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)

x = torch.zeros(4, 3, 32, 32)

# buraya yaz

Sample solution

model.py
import torch
from torch import nn

torch.manual_seed(0)
govde = nn.Sequential(
    nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
    nn.AdaptiveAvgPool2d(1), nn.Flatten())
bas = nn.Linear(32, 1000)
onceden = nn.Sequential(govde, bas)

x = torch.zeros(4, 3, 32, 32)

yeni = nn.Sequential(govde, nn.Linear(bas.in_features, 5))
print(onceden(x).shape)
print(yeni(x).shape)

Expected output

torch.Size([4, 1000])
torch.Size([4, 5])

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 In which situation does transfer learning make the most difference?

  • When the new task has little labelled data Correct answer
  • When there are millions of labelled examples
  • When the data is tabular
  • Never

Explanation: Pre-learned general features provide knowledge that couldn't be learned from little data.

2 Replace the head layer for the new task.

Question 
model.fc = nn.Linear(model.fc.____, 5)
Answer 
model.fc = nn.Linear(model.fc.in_features, 5)

Explanation: in_features is the old head's input size; the body's output.

3 Match the concepts.

Term Meaning
body General features
head Task-specific classifier
pretrained Learned on large data
in_features Body output size

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4