CodeBlanks Logo Icon CodeBlanks
Login Register

Summary - Autograd

In this group you learned autograd, requires_grad, and backward().

Summary: autograd

In this group:

  • Tensors with requires_grad=True are tracked; derived tensors are tracked too.
  • backward() computes gradients from a scalar result and fills the leaves' .grad.
  • Gradients accumulate; they must be zeroed every step.
  • no_grad() and detach() stop tracking.
  • The computational graph is built dynamically; grad_fn shows the nodes, the chain rule is applied backwards.

Practice Gradient descent by hand

Minimise kayip = (w - 3) ** 2 with 50 steps of gradient descent starting from w = 0 with a learning rate of 0.1: at each step compute the loss, call backward(), update inside torch.no_grad() and zero the gradient. Print the final w with round(..., 3).

Starter code

model.py
import torch

w = torch.tensor(0.0, requires_grad=True)

# buraya yaz

Sample solution

model.py
import torch

w = torch.tensor(0.0, requires_grad=True)

for _ in range(50):
    kayip = (w - 3) ** 2
    kayip.backward()
    with torch.no_grad():
        w -= 0.1 * w.grad
    w.grad.zero_()
print(round(w.item(), 3))

Expected output

3.0

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 Why is the update in gradient descent with autograd done inside <code>torch.no_grad()</code>?

  • It isn't required
  • To zero the gradient
  • So the update itself isn't recorded in the graph Correct answer
  • Because it is more accurate

Explanation: w -= lr·grad is a parameter adjustment, not a computation to differentiate.

2 Do the update without tracking.

Question 
with torch.no_grad():
    w ____ 0.1 * w.grad
Answer 
with torch.no_grad():
    w -= 0.1 * w.grad

Explanation: A step is taken in the opposite direction of the gradient.

3 Match the tools.

Term Meaning
backward() Gradient computation
.grad Gradient value
zero_() Zeroing
no_grad() Untracked block

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4