Machine Learning
What is RNN?
RNN is a model designed to process sequential data.
What is an RNN?
A recurrent neural network (RNN) processes sequences (text, time series, audio) step by step. At each step it combines the new input with the hidden state coming from the previous step; that way it carries a summary of the past:
# h_t = tanh(W_x · x_t + W_h · h_(t-1) + b)
h = torch.zeros(gizli)
for x_t in dizi:
h = torch.tanh(W_x @ x_t + W_h @ h + b)
The same weights are used at every step; this is the time-axis counterpart of weight sharing in CNNs and lets sequences of any length be processed.
Sequence problems
RNNs fit different input-output forms:
- Many to one: sentiment analysis (sentence → positive/negative).
- Many to many: tagging (a part of speech for every word), predicting every step of a time series.
- One to many: text generation (start → sentence).
Sequential processing is also the RNN's weakness: steps can't be computed in parallel and early information gets lost in long sequences. The transformer was developed to solve these two problems.
Exercise An RNN by hand
Run an RNN with hidden size 2 by hand: start h at zero and at every step set h = torch.tanh(W_x @ x_t + W_h @ h). After the three inputs of the sequence, print h as a list rounded with round(..., 3).
Show example solution
import torch
from torch import nn
W_x = torch.tensor([[1.0], [0.5]])
W_h = torch.tensor([[0.5, 0.0], [0.0, 0.5]])
dizi = [torch.tensor([1.0]), torch.tensor([0.0]), torch.tensor([-1.0])]
h = torch.zeros(2)
for x_t in dizi:
h = torch.tanh(W_x @ x_t + W_h @ h)
print([round(v, 3) for v in h.tolist()])
Test yourself
-
What does using the same weights at every step allow in an RNN?
- Learning without labels
- Processing sequences of different lengths
- Parallel training
- More parameters
Show answer
Processing sequences of different lengths — The parameter count doesn't depend on the sequence length.
-
Update the hidden state.
h = torch.____(W_x @ x_t + W_h @ h + b)
Show answer
h = torch.tanh(W_x @ x_t + W_h @ h + b)
-
Match the problems with the forms.
- Sentiment analysis
- Many to one
- Part-of-speech tagging
- Many to many
- Text generation
- One to many
- hidden state
- Summary of the past
Next lesson: Hidden state
