backward()
- Machine Learning course
- Advanced
- Lesson 38/78
- 5 min
The backward() method computes gradients from the loss.
backward() in training
In training, kayip.backward() fills the .grad of every parameter in the model. Gradient sizes say something about the health of training: if very small, learning has stalled (vanishing gradient); if very large, there is a risk of explosion:
kayip.backward()
for ad, p in model.named_parameters():
print(ad, p.grad.norm().item())
norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)clip_grad_norm_ limits the total gradient norm; it prevents explosions especially when training RNNs and transformers.
Gradient accumulation
If memory isn't enough for a large batch, the gradients of a few small batches can be accumulated and applied in one step. Gradient accumulation is used deliberately here:
biriktir = 4
for i, (xb, yb) in enumerate(yukleyici):
kayip = kayip_fn(model(xb), yb) / biriktir # keep the mean
kayip.backward()
if (i + 1) % biriktir == 0:
opt.step()
opt.zero_grad()The result is roughly equal to training with a batch four times larger.
Practice Gradient clipping
Do one forward and backward pass with the given model and data. Call clip_grad_norm_(model.parameters(), max_norm=0.5); print on one line that the returned (pre-clipping) norm is above 0.5 and that the total norm after clipping doesn't exceed 0.5 (<= 0.5 + 1e-6).
Starter code
import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(0)
X = torch.randn(200, 2)
y = (X[:, 0] + X[:, 1] > 0).long()
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 2))
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.SGD(model.parameters(), lr=0.1)
with torch.no_grad():
model[2].weight.mul_(20)
# buraya yazSample solution
import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(0)
X = torch.randn(200, 2)
y = (X[:, 0] + X[:, 1] > 0).long()
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 2))
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.SGD(model.parameters(), lr=0.1)
with torch.no_grad():
model[2].weight.mul_(20)
kayip_fn(model(X), y).backward()
once = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=0.5)
sonra = torch.sqrt(sum((p.grad ** 2).sum() for p in model.parameters()))
print(once.item() > 0.5, sonra.item() <= 0.5 + 1e-6)Expected output
True True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 When should gradient clipping be called?
- Before the forward pass
- After training
- Between backward() and step() Correct answer
- Before zero_grad()
Explanation: The gradients are limited after they are computed and before they are used.
2 Limit the total gradient norm to 1.
torch.nn.utils.____(model.parameters(), max_norm=1.0)torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)Explanation: clip_grad_norm_ scales all gradients together.
3 Match the concepts.
| Term | Meaning |
|---|---|
p.grad.norm() |
Gradient size |
vanishing gradient |
Learning stalls |
clip_grad_norm_ |
Prevents explosion |
accumulation |
Virtual large batch |