What is an optimizer?

An optimizer updates model weights using gradients.

What is an optimiser?

An optimiser is the algorithm that updates the parameters using the gradients. They all share the same interface: parameters and settings at construction, zero_grad() and step() in training. Their difference is how they turn the gradient into a step:

torch.optim.SGD(params, lr=0.1)                    # p -= lr·g
torch.optim.SGD(params, lr=0.1, momentum=0.9)      # accumulated velocity
torch.optim.Adam(params, lr=1e-3)                  # an adaptive step per parameter
torch.optim.AdamW(params, lr=1e-3, weight_decay=0.01)

The loss surface isn't a smooth bowl: it contains narrow valleys, plateaus and saddle points. A good optimiser moves fast and stably through them.

Gradient descent by hand

To see by hand what an optimiser does, let's minimise f(w) = (w - 3)². The derivative is 2(w - 3); at every step the error shrinks by a factor of (1 - 2·lr):

w, lr = 0.0, 0.1
for adim in range(3):
    g = 2 * (w - 3)
    w -= lr * g
    print(round(w, 3))   # 0.6, 1.08, 1.464

Each step covers 20% of the remaining distance to the optimum 3.

Exercise By hand and with the optimiser

For f(w) = (w - 3)², take 3 steps with torch.optim.SGD([w], lr=0.1); after every step print w with round(..., 3) on separate lines. The results should match those computed by hand on the card.

Show example solution

import torch

w = torch.tensor(0.0, requires_grad=True)

opt = torch.optim.SGD([w], lr=0.1)
for _ in range(3):
    opt.zero_grad()
    ((w - 3) ** 2).backward()
    opt.step()
    print(round(w.item(), 3))

Test yourself

Which two methods do all PyTorch optimisers share?
- fit() and predict()
- forward() and backward()
- train() and eval()
- zero_grad() and step()
Show answer

zero_grad() and step() — The training loop calls only these two on the optimiser.

Set up the SGD optimiser.

opt = torch.optim.____([w], lr=0.1)

Show answer

opt = torch.optim.SGD([w], lr=0.1)

Match the optimisers.

SGD

p -= lr·g
momentum

Accumulated velocity
Adam

Adaptive step
AdamW

Decoupled weight decay

Next lesson: SGD

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies