CodeBlanks Logo Icon CodeBlanks
Login Register

Summary - Optimization

In this group you learned SGD, Adam, and learning rate.

Summary: optimisation

In this group:

  • Optimisers turn the gradient into a step; the interface is zero_grad() + step().
  • SGD is p -= lr·g; momentum accumulates consistent directions.
  • Adam takes an adaptive per-parameter step with the m and v averages; the first step ≈ lr, scale-independent.
  • The learning rate is the most important setting; warm-up, cosine and OneCycle schedulers.
  • The position of zero_grad, weight_decay and AdamW.

Practice A badly scaled loss

Minimise f(w) = 100·(w₀ - 1)² + (w₁ - 2)² from w = [0, 0] in 100 steps: once with SGD(lr=0.009), once with Adam(lr=0.1). For each optimiser print the final w as a list rounded with round(..., 2) on separate lines (SGD first).

Starter code

model.py
import torch

# buraya yaz

Sample solution

model.py
import torch

for yap in [lambda p: torch.optim.SGD(p, lr=0.009), lambda p: torch.optim.Adam(p, lr=0.1)]:
    w = torch.zeros(2, requires_grad=True)
    opt = yap([w])
    for _ in range(100):
        opt.zero_grad()
        (100 * (w[0] - 1) ** 2 + (w[1] - 2) ** 2).backward()
        opt.step()
    print([round(v, 2) for v in w.tolist()])

Expected output

[1.0, 1.67]
[1.0, 2.01]

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 On a badly scaled loss (very steep in one direction), which usually progresses faster?

  • Both the same
  • Neither
  • Adam Correct answer
  • SGD without momentum

Explanation: Adam normalises each direction by its own gradient scale.

2 Set up Adam with decoupled weight decay.

Question 
opt = torch.optim.____(model.parameters(), lr=1e-3, weight_decay=0.01)
Answer 
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)

Explanation: AdamW separates weight decay from the adaptive step.

3 Match the optimisers with the properties.

Term Meaning
SGD Simple, proportional to gradient
SGD + momentum Velocity accumulation
Adam Adaptive, scale-independent
AdamW Transformer default

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4