Machine Learning
Adam
Adam updates gradients with adaptive learning rates.
Adam
Adam keeps, for every parameter, a moving average of the gradient (m, momentum) and of its square (v); it scales the step by m / √v. That way every parameter takes a step adapted to its own scale:
# m = β1·m + (1-β1)·g (β1 = 0.9)
# v = β2·v + (1-β2)·g² (β2 = 0.999)
# m̂ = m / (1-β1^t), v̂ = v / (1-β2^t) bias correction
# p -= lr · m̂ / (√v̂ + ε)
Because Adam works well on most problems with little tuning, it is the most common default in deep learning.
Scale independence
In SGD the step is proportional to the gradient: multiplying the loss by 1000 makes the step 1000 times larger. In Adam the ratio stays constant:
for olcek in [1, 1000]:
w = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.Adam([w], lr=0.1)
(olcek * (w - 3) ** 2).backward()
opt.step()
print(olcek, round(w.item(), 4)) # 0.1 for both
This property makes Adam stable when the gradients of different layers are of very different sizes.
Exercise Adam's first step
For olcek 1 and 1000, take a single Adam step (lr=0.1) with the loss (olcek * (w - 3) ** 2) and print w with round(..., 4). Then do the same experiment with SGD(lr=0.0001) and print the two w values on one line.
Show example solution
import torch
for olcek in [1, 1000]:
w = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.Adam([w], lr=0.1)
(olcek * (w - 3) ** 2).backward()
opt.step()
print(olcek, round(w.item(), 4))
sgd = []
for olcek in [1, 1000]:
w = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.SGD([w], lr=0.0001)
(olcek * (w - 3) ** 2).backward()
opt.step()
sgd.append(round(w.item(), 4))
print(*sgd)
Test yourself
-
Roughly how big is Adam's first step?
- The gradient itself
- lr (independent of the gradient's scale)
- lr · gradient
- Zero
Show answer
lr (independent of the gradient's scale) — At the first step m̂ = g and √v̂ = |g|; the ratio reduces to the sign.
-
Give Adam's beta values.
opt = torch.optim.Adam(params, lr=1e-3, ____=(0.9, 0.999))
Show answer
opt = torch.optim.Adam(params, lr=1e-3, betas=(0.9, 0.999))
-
Match the pieces.
- m
- Gradient average
- v
- Square average
- bias correction
- Balances the first steps
- ε
- Prevents division by zero
Next lesson: Learning rate
