requires_grad
- Machine Learning course
- Advanced
- Lesson 20/78
- 5 min
requires_grad=True enables gradient tracking for a tensor.
requires_grad
Autograd only tracks tensors with requires_grad=True. Model parameters (nn.Linear weights) turn this on automatically; it is off for input data and labels. Every tensor derived from such a tensor is tracked too:
w = torch.tensor([1.0, 2.0], requires_grad=True)
x = torch.tensor([3.0, 4.0]) # not tracked
z = w * x
print(z.requires_grad) # True: derived from w
print(x.requires_grad) # FalseTo turn it on later, the in-place version is used: t.requires_grad_().
no_grad and detach
Gradients aren't needed during evaluation and prediction; turning tracking off saves memory and time. torch.no_grad() stops tracking in a block, detach() on a single tensor:
with torch.no_grad():
tahmin = model(X_test) # no graph is built
print(tahmin.requires_grad) # False
z = (w * 2).detach() # a copy cut off from w
print(z.requires_grad) # FalseTo turn an output into NumPy, a tracked tensor must also be detach()ed first.
Practice Control tracking
w is a tracked tensor, x an untracked one. Print the requires_grad values of w * x, of w * x computed inside torch.no_grad() and of (w * x).detach() on one line.
Starter code
import torch
w = torch.tensor([1.0, 2.0], requires_grad=True)
x = torch.tensor([3.0, 4.0])
# buraya yazSample solution
import torch
w = torch.tensor([1.0, 2.0], requires_grad=True)
x = torch.tensor([3.0, 4.0])
a = w * x
with torch.no_grad():
b = w * x
c = (w * x).detach()
print(a.requires_grad, b.requires_grad, c.requires_grad)Expected output
True False False
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What happens if <code>requires_grad=True</code> is given on an integer tensor?
- It becomes float automatically
- The gradient is zero
- It errors; only floating types are differentiable Correct answer
- It works fine
Explanation: Derivatives need continuous values; int tensors can't be tracked.
2 Turn tracking off during prediction.
with torch.____():
tahmin = model(X_test)with torch.no_grad():
tahmin = model(X_test)Explanation: torch.no_grad() stops the graph from being built throughout the block.
3 Match the situations.
| Term | Meaning |
|---|---|
requires_grad=True |
Parameter |
girdi verisi |
Not tracked |
torch.no_grad() |
No tracking in a block |
detach() |
Cut from the graph |