CodeBlanks Logo Icon CodeBlanks
Login Register

Summary - RNN

In this group you learned RNN, hidden state, and vanishing gradients.

Summary: RNN

In this group:

  • An RNN processes a sequence step by step: h_t = tanh(W_x x_t + W_h h_(t-1) + b); the weights are shared across steps.
  • nn.RNN: output (N, T, H), h_n (layer, N, H); batch_first=True.
  • Different lengths: pad_sequence, the real length, pack_padded_sequence.
  • In long sequences the gradient vanishes or explodes; clipping and gated architectures are the fixes.
  • nn.Embedding turns word indices into learned vectors.

Practice A two-layer RNN

Apply an nn.RNN(16, 32, num_layers=2, batch_first=True) layer to a (4, 10, 16) input. Print the output and h_n shapes on one line, then the parameter count (the second layer's input is 32).

Starter code

model.py
import torch
from torch import nn

x = torch.zeros(4, 10, 16)

# buraya yaz

Sample solution

model.py
import torch
from torch import nn

x = torch.zeros(4, 10, 16)

rnn = nn.RNN(16, 32, num_layers=2, batch_first=True)
output, h_n = rnn(x)
print(output.shape, h_n.shape)
print(sum(p.numel() for p in rnn.parameters()))

Expected output

torch.Size([4, 10, 32]) torch.Size([2, 4, 32])
3712

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is the RNN's main weakness compared with the transformer?

  • Steps are processed sequentially and can't be computed in parallel Correct answer
  • It has no parameters
  • It can't process text
  • It can't use embeddings

Explanation: Because each step waits for the previous one, exploiting GPU parallelism is hard.

2 Build a two-layer RNN.

Question 
rnn = nn.RNN(16, 32, ____=2, batch_first=True)
Answer 
rnn = nn.RNN(16, 32, num_layers=2, batch_first=True)

Explanation: num_layers stacks RNN layers.

3 Match the concepts.

Term Meaning
hidden state Summary of the past
padding Equal length
vanishing gradient Long-range problem
embedding Word vectors

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4