Machine Learning
parameters()
The parameters() method returns the model's learnable parameters.
parameters()
parameters() walks all the learned tensors in the module; this is what is given to the optimiser. named_parameters() returns them with their names:
model = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 3))
for ad, p in model.named_parameters():
print(ad, tuple(p.shape))
# 0.weight (8, 4) 0.bias (8,) 2.weight (3, 8) 2.bias (3,)
print(sum(p.numel() for p in model.parameters())) # 67
Because ReLU has no parameters, the name with number 1 doesn't appear in the list.
Freezing and state_dict
If a parameter's requires_grad is turned off, that parameter is frozen: it gets no gradient and isn't updated. In transfer learning pretrained layers are protected this way. state_dict() collects all parameters and buffers by name in a dict; it is the standard way to save a model:
for p in model[0].parameters():
p.requires_grad = False # freeze the first layer
egitilebilir = sum(p.numel() for p in model.parameters() if p.requires_grad)
torch.save(model.state_dict(), "model.pt")
model.load_state_dict(torch.load("model.pt"))
Exercise Count and freeze parameters
For nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 3)), print the parameter names as a list and the total parameter count on separate lines. Then freeze the first layer and print the number of trainable parameters.
Show example solution
import torch
from torch import nn
model = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 3))
print([ad for ad, _ in model.named_parameters()])
print(sum(p.numel() for p in model.parameters()))
for p in model[0].parameters():
p.requires_grad = False
print(sum(p.numel() for p in model.parameters() if p.requires_grad))
Test yourself
-
How many parameters does <code>nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 1))</code> contain?
Show answer
61 — 10·5 + 5 = 55; 5·1 + 1 = 6; 61 in total.
-
Walk the parameters with their names.
for ad, p in model.____():
Show answer
for ad, p in model.named_parameters():
-
Match the methods.
- parameters()
- Learned tensors
- named_parameters()
- With their names
- state_dict()
- Dict to save
- requires_grad = False
- Freezing
Next lesson: Summary - nn.Module
