Learning rate
- Machine Learning course
- Advanced
- Lesson 72/78
- 5 min
Learning rate determines how much weights change each step.
The learning rate
The learning rate is the most important hyperparameter of deep learning:
- Too small: the loss falls very slowly.
- Right: the loss falls fast and stably.
- Too large: the loss oscillates or explodes (NaN).
# for f(w) = (w - 3)² the error is multiplied by (1 - 2·lr) each step
# lr = 0.01 -> 0.98 (slow)
# lr = 0.1 -> 0.8 (good)
# lr = 1.1 -> -1.2 (|·| > 1: diverges)In practice a few values are first tried on a logarithmic scale: 1e-4, 1e-3, 1e-2.
Warm-up and cosine decay
Modern training usually raises the learning rate from zero in the first steps (warm-up), then slowly decreases it along a cosine curve. Warm-up prevents large steps while Adam's statistics are still unstable:
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
zam = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=1e-3, total_steps=1000)
for adim in range(1000):
egit_bir_adim()
zam.step() # every step (not every epoch)OneCycleLR combines warm-up and decay in a single scheduler.
Practice Compare the rates
For lr = 0.01, 0.1 and 1.1, apply 20 steps of SGD on f(w) = (w - 3)² starting from w = 0. On each line print lr and the final w with round(..., 3).
Starter code
import torch
# buraya yazSample solution
import torch
for lr in [0.01, 0.1, 1.1]:
w = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.SGD([w], lr=lr)
for _ in range(20):
opt.zero_grad()
((w - 3) ** 2).backward()
opt.step()
print(lr, round(w.item(), 3))Expected output
0.01 0.997 0.1 2.965 1.1 -112.013
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If the loss suddenly becomes <code>nan</code> during training, what is the first suspect?
- The learning rate being too large Correct answer
- Too little data
- The model being too small
- The test set
Explanation: Large steps blow up the parameters; lower the rate or clip the gradients.
2 Set up cosine-shaped decay.
zam = torch.optim.lr_scheduler.____(opt, T_max=100)zam = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=100)Explanation: CosineAnnealingLR lowers the rate along a cosine over T_max steps.
3 Match the behaviours.
| Term | Meaning |
|---|---|
lr too small |
Slow decrease |
lr too large |
Explosion, NaN |
warm-up |
Small steps at first |
cosine decay |
Fine-tuning at the end |