requires_grad

requires_grad=True enables gradient tracking for a tensor.

requires_grad

Autograd only tracks tensors with requires_grad=True. Model parameters (nn.Linear weights) turn this on automatically; it is off for input data and labels. Every tensor derived from such a tensor is tracked too:

w = torch.tensor([1.0, 2.0], requires_grad=True)
x = torch.tensor([3.0, 4.0])          # not tracked
z = w * x
print(z.requires_grad)                # True: derived from w
print(x.requires_grad)                # False

To turn it on later, the in-place version is used: t.requires_grad_().

no_grad and detach

Gradients aren't needed during evaluation and prediction; turning tracking off saves memory and time. torch.no_grad() stops tracking in a block, detach() on a single tensor:

with torch.no_grad():
    tahmin = model(X_test)        # no graph is built
print(tahmin.requires_grad)       # False
z = (w * 2).detach()               # a copy cut off from w
print(z.requires_grad)            # False

To turn an output into NumPy, a tracked tensor must also be detach()ed first.

Exercise Control tracking

w is a tracked tensor, x an untracked one. Print the requires_grad values of w * x, of w * x computed inside torch.no_grad() and of (w * x).detach() on one line.

Show example solution

import torch

w = torch.tensor([1.0, 2.0], requires_grad=True)
x = torch.tensor([3.0, 4.0])

a = w * x
with torch.no_grad():
    b = w * x
c = (w * x).detach()
print(a.requires_grad, b.requires_grad, c.requires_grad)

Test yourself

What happens if <code>requires_grad=True</code> is given on an integer tensor?
- It becomes float automatically
- The gradient is zero
- It errors; only floating types are differentiable
- It works fine
Show answer

It errors; only floating types are differentiable — Derivatives need continuous values; int tensors can't be tracked.

Turn tracking off during prediction.

with torch.____():
    tahmin = model(X_test)

Show answer

with torch.no_grad():
    tahmin = model(X_test)

Match the situations.

requires_grad=True

Parameter
girdi verisi

Not tracked
torch.no_grad()

No tracking in a block
detach()

Cut from the graph

Next lesson: backward()

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies