Summary - RNN

In this group you learned RNN, hidden state, and vanishing gradients.

Summary: RNN

In this group:

An RNN processes a sequence step by step: h_t = tanh(W_x x_t + W_h h_(t-1) + b); the weights are shared across steps.
nn.RNN: output (N, T, H), h_n (layer, N, H); batch_first=True.
Different lengths: pad_sequence, the real length, pack_padded_sequence.
In long sequences the gradient vanishes or explodes; clipping and gated architectures are the fixes.
nn.Embedding turns word indices into learned vectors.

Exercise A two-layer RNN

Apply an nn.RNN(16, 32, num_layers=2, batch_first=True) layer to a (4, 10, 16) input. Print the output and h_n shapes on one line, then the parameter count (the second layer's input is 32).

Show example solution

import torch
from torch import nn

x = torch.zeros(4, 10, 16)

rnn = nn.RNN(16, 32, num_layers=2, batch_first=True)
output, h_n = rnn(x)
print(output.shape, h_n.shape)
print(sum(p.numel() for p in rnn.parameters()))

Test yourself

What is the RNN's main weakness compared with the transformer?
- Steps are processed sequentially and can't be computed in parallel
- It has no parameters
- It can't process text
- It can't use embeddings
Show answer

Steps are processed sequentially and can't be computed in parallel — Because each step waits for the previous one, exploiting GPU parallelism is hard.

Build a two-layer RNN.

rnn = nn.RNN(16, 32, ____=2, batch_first=True)

Show answer

rnn = nn.RNN(16, 32, num_layers=2, batch_first=True)

Match the concepts.

hidden state

Summary of the past
padding

Equal length
vanishing gradient

Long-range problem
embedding

Word vectors

Next lesson: What is LSTM?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies