What is an optimizer?
- Machine Learning course
- Advanced
- Lesson 69/78
- 5 min
An optimizer updates model weights using gradients.
What is an optimiser?
An optimiser is the algorithm that updates the parameters using the gradients. They all share the same interface: parameters and settings at construction, zero_grad() and step() in training. Their difference is how they turn the gradient into a step:
torch.optim.SGD(params, lr=0.1) # p -= lr·g
torch.optim.SGD(params, lr=0.1, momentum=0.9) # accumulated velocity
torch.optim.Adam(params, lr=1e-3) # an adaptive step per parameter
torch.optim.AdamW(params, lr=1e-3, weight_decay=0.01)The loss surface isn't a smooth bowl: it contains narrow valleys, plateaus and saddle points. A good optimiser moves fast and stably through them.
Gradient descent by hand
To see by hand what an optimiser does, let's minimise f(w) = (w - 3)². The derivative is 2(w - 3); at every step the error shrinks by a factor of (1 - 2·lr):
w, lr = 0.0, 0.1
for adim in range(3):
g = 2 * (w - 3)
w -= lr * g
print(round(w, 3)) # 0.6, 1.08, 1.464Each step covers 20% of the remaining distance to the optimum 3.
Practice By hand and with the optimiser
For f(w) = (w - 3)², take 3 steps with torch.optim.SGD([w], lr=0.1); after every step print w with round(..., 3) on separate lines. The results should match those computed by hand on the card.
Starter code
import torch
w = torch.tensor(0.0, requires_grad=True)
# buraya yazSample solution
import torch
w = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.SGD([w], lr=0.1)
for _ in range(3):
opt.zero_grad()
((w - 3) ** 2).backward()
opt.step()
print(round(w.item(), 3))Expected output
0.6 1.08 1.464
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Which two methods do all PyTorch optimisers share?
- fit() and predict()
- forward() and backward()
- train() and eval()
- zero_grad() and step() Correct answer
Explanation: The training loop calls only these two on the optimiser.
2 Set up the SGD optimiser.
opt = torch.optim.____([w], lr=0.1)opt = torch.optim.SGD([w], lr=0.1)Explanation: SGD is stochastic gradient descent.
3 Match the optimisers.
| Term | Meaning |
|---|---|
SGD |
p -= lr·g |
momentum |
Accumulated velocity |
Adam |
Adaptive step |
AdamW |
Decoupled weight decay |