CodeBlanks Logo Icon CodeBlanks
Login Register

What is autograd?

Autograd automatically computes gradients of tensor operations.

What is autograd?

Training neural networks requires knowing the derivative (gradient) of the loss with respect to every parameter. Autograd records the operations performed on tensors and computes these derivatives automatically by applying the chain rule:

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2 + 2 * x      # y = x² + 2x
y.backward()            # compute dy/dx
print(x.grad.item())    # 8.0  (2x + 2, x = 3)

Writing derivatives by hand is impossible in networks with millions of parameters; autograd is the core tool that makes deep learning practical.

What does the gradient tell?

The gradient points in the direction in which the loss increases fastest. Moving the parameter a small step in the opposite direction of the gradient reduces the loss; this is the essence of gradient descent:

w = torch.tensor(5.0, requires_grad=True)
kayip = (w - 2) ** 2       # minimum at w = 2
kayip.backward()
print(w.grad.item())       # 6.0: positive, decrease w
with torch.no_grad():
    w -= 0.1 * w.grad      # w = 4.4, closer to 2

When this loop is repeated thousands of times, the parameters move towards the values that minimise the loss.

Practice First derivative

For x = torch.tensor(2.0, requires_grad=True), compute y = x ** 3 - 4 * x and call backward(). Print y.item() and x.grad.item() on one line.

Starter code

model.py
import torch

# buraya yaz

Sample solution

model.py
import torch

x = torch.tensor(2.0, requires_grad=True)
y = x ** 3 - 4 * x
y.backward()
print(y.item(), x.grad.item())

Expected output

0.0 8.0

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 For <code>y = 3 * x ** 2</code>, what is <code>x.grad</code> at x = 2?

  • 6
  • 24
  • 3
  • 12 Correct answer

Explanation: dy/dx = 6x; 12 at x = 2.

2 Have the derivatives computed.

Question 
kayip.____()
Answer 
kayip.backward()

Explanation: backward() computes all gradients backwards from the loss.

3 Match the concepts.

Term Meaning
requires_grad Its derivative is tracked
backward() Compute gradients
x.grad Computed derivative
gradient descent Step in the opposite direction

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4