Summary - Optimization

In this group you learned SGD, Adam, and learning rate.

Summary: optimisation

In this group:

Optimisers turn the gradient into a step; the interface is zero_grad() + step().
SGD is p -= lr·g; momentum accumulates consistent directions.
Adam takes an adaptive per-parameter step with the m and v averages; the first step ≈ lr, scale-independent.
The learning rate is the most important setting; warm-up, cosine and OneCycle schedulers.
The position of zero_grad, weight_decay and AdamW.

Exercise A badly scaled loss

Minimise f(w) = 100·(w₀ - 1)² + (w₁ - 2)² from w = [0, 0] in 100 steps: once with SGD(lr=0.009), once with Adam(lr=0.1). For each optimiser print the final w as a list rounded with round(..., 2) on separate lines (SGD first).

Show example solution

import torch

for yap in [lambda p: torch.optim.SGD(p, lr=0.009), lambda p: torch.optim.Adam(p, lr=0.1)]:
    w = torch.zeros(2, requires_grad=True)
    opt = yap([w])
    for _ in range(100):
        opt.zero_grad()
        (100 * (w[0] - 1) ** 2 + (w[1] - 2) ** 2).backward()
        opt.step()
    print([round(v, 2) for v in w.tolist()])

Test yourself

On a badly scaled loss (very steep in one direction), which usually progresses faster?
- Both the same
- Neither
- Adam
- SGD without momentum
Show answer

Adam — Adam normalises each direction by its own gradient scale.

Set up Adam with decoupled weight decay.

opt = torch.optim.____(model.parameters(), lr=1e-3, weight_decay=0.01)

Show answer

opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)

Match the optimisers with the properties.

SGD

Simple, proportional to gradient
SGD + momentum

Velocity accumulation
Adam

Adaptive, scale-independent
AdamW

Transformer default

Next lesson: Challenge 1

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies