Machine Learning
backward()
The backward() method computes gradients from the loss.
backward() in training
In training, kayip.backward() fills the .grad of every parameter in the model. Gradient sizes say something about the health of training: if very small, learning has stalled (vanishing gradient); if very large, there is a risk of explosion:
kayip.backward()
for ad, p in model.named_parameters():
print(ad, p.grad.norm().item())
norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
clip_grad_norm_ limits the total gradient norm; it prevents explosions especially when training RNNs and transformers.
Gradient accumulation
If memory isn't enough for a large batch, the gradients of a few small batches can be accumulated and applied in one step. Gradient accumulation is used deliberately here:
biriktir = 4
for i, (xb, yb) in enumerate(yukleyici):
kayip = kayip_fn(model(xb), yb) / biriktir # keep the mean
kayip.backward()
if (i + 1) % biriktir == 0:
opt.step()
opt.zero_grad()
The result is roughly equal to training with a batch four times larger.
Exercise Gradient clipping
Do one forward and backward pass with the given model and data. Call clip_grad_norm_(model.parameters(), max_norm=0.5); print on one line that the returned (pre-clipping) norm is above 0.5 and that the total norm after clipping doesn't exceed 0.5 (<= 0.5 + 1e-6).
Show example solution
import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(0)
X = torch.randn(200, 2)
y = (X[:, 0] + X[:, 1] > 0).long()
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 2))
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.SGD(model.parameters(), lr=0.1)
with torch.no_grad():
model[2].weight.mul_(20)
kayip_fn(model(X), y).backward()
once = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=0.5)
sonra = torch.sqrt(sum((p.grad ** 2).sum() for p in model.parameters()))
print(once.item() > 0.5, sonra.item() <= 0.5 + 1e-6)
Test yourself
-
When should gradient clipping be called?
- Before the forward pass
- After training
- Between backward() and step()
- Before zero_grad()
Show answer
Between backward() and step() — The gradients are limited after they are computed and before they are used.
-
Limit the total gradient norm to 1.
torch.nn.utils.____(model.parameters(), max_norm=1.0)
Show answer
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
-
Match the concepts.
- p.grad.norm()
- Gradient size
- vanishing gradient
- Learning stalls
- clip_grad_norm_
- Prevents explosion
- accumulation
- Virtual large batch
Next lesson: optimizer.step()
