Adam
- Machine Learning course
- Advanced
- Lesson 71/78
- 5 min
Adam updates gradients with adaptive learning rates.
Adam
Adam keeps, for every parameter, a moving average of the gradient (m, momentum) and of its square (v); it scales the step by m / √v. That way every parameter takes a step adapted to its own scale:
# m = β1·m + (1-β1)·g (β1 = 0.9)
# v = β2·v + (1-β2)·g² (β2 = 0.999)
# m̂ = m / (1-β1^t), v̂ = v / (1-β2^t) bias correction
# p -= lr · m̂ / (√v̂ + ε)Because Adam works well on most problems with little tuning, it is the most common default in deep learning.
Scale independence
In SGD the step is proportional to the gradient: multiplying the loss by 1000 makes the step 1000 times larger. In Adam the ratio stays constant:
for olcek in [1, 1000]:
w = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.Adam([w], lr=0.1)
(olcek * (w - 3) ** 2).backward()
opt.step()
print(olcek, round(w.item(), 4)) # 0.1 for bothThis property makes Adam stable when the gradients of different layers are of very different sizes.
Practice Adam's first step
For olcek 1 and 1000, take a single Adam step (lr=0.1) with the loss (olcek * (w - 3) ** 2) and print w with round(..., 4). Then do the same experiment with SGD(lr=0.0001) and print the two w values on one line.
Starter code
import torch
# buraya yazSample solution
import torch
for olcek in [1, 1000]:
w = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.Adam([w], lr=0.1)
(olcek * (w - 3) ** 2).backward()
opt.step()
print(olcek, round(w.item(), 4))
sgd = []
for olcek in [1, 1000]:
w = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.SGD([w], lr=0.0001)
(olcek * (w - 3) ** 2).backward()
opt.step()
sgd.append(round(w.item(), 4))
print(*sgd)Expected output
1 0.1 1000 0.1 0.0006 0.6
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Roughly how big is Adam's first step?
- The gradient itself
- lr (independent of the gradient's scale) Correct answer
- lr · gradient
- Zero
Explanation: At the first step m̂ = g and √v̂ = |g|; the ratio reduces to the sign.
2 Give Adam's beta values.
opt = torch.optim.Adam(params, lr=1e-3, ____=(0.9, 0.999))opt = torch.optim.Adam(params, lr=1e-3, betas=(0.9, 0.999))Explanation: betas is the (β1, β2) pair.
3 Match the pieces.
| Term | Meaning |
|---|---|
m |
Gradient average |
v |
Square average |
bias correction |
Balances the first steps |
ε |
Prevents division by zero |