What is RNN?
- Machine Learning course
- Advanced
- Lesson 47/78
- 5 min
RNN is a model designed to process sequential data.
What is an RNN?
A recurrent neural network (RNN) processes sequences (text, time series, audio) step by step. At each step it combines the new input with the hidden state coming from the previous step; that way it carries a summary of the past:
# h_t = tanh(W_x · x_t + W_h · h_(t-1) + b)
h = torch.zeros(gizli)
for x_t in dizi:
h = torch.tanh(W_x @ x_t + W_h @ h + b)The same weights are used at every step; this is the time-axis counterpart of weight sharing in CNNs and lets sequences of any length be processed.
Sequence problems
RNNs fit different input-output forms:
- Many to one: sentiment analysis (sentence → positive/negative).
- Many to many: tagging (a part of speech for every word), predicting every step of a time series.
- One to many: text generation (start → sentence).
Sequential processing is also the RNN's weakness: steps can't be computed in parallel and early information gets lost in long sequences. The transformer was developed to solve these two problems.
Practice An RNN by hand
Run an RNN with hidden size 2 by hand: start h at zero and at every step set h = torch.tanh(W_x @ x_t + W_h @ h). After the three inputs of the sequence, print h as a list rounded with round(..., 3).
Starter code
import torch
from torch import nn
W_x = torch.tensor([[1.0], [0.5]])
W_h = torch.tensor([[0.5, 0.0], [0.0, 0.5]])
dizi = [torch.tensor([1.0]), torch.tensor([0.0]), torch.tensor([-1.0])]
# buraya yazSample solution
import torch
from torch import nn
W_x = torch.tensor([[1.0], [0.5]])
W_h = torch.tensor([[0.5, 0.0], [0.0, 0.5]])
dizi = [torch.tensor([1.0]), torch.tensor([0.0]), torch.tensor([-1.0])]
h = torch.zeros(2)
for x_t in dizi:
h = torch.tanh(W_x @ x_t + W_h @ h)
print([round(v, 3) for v in h.tolist()])Expected output
[-0.674, -0.368]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What does using the same weights at every step allow in an RNN?
- Learning without labels
- Processing sequences of different lengths Correct answer
- Parallel training
- More parameters
Explanation: The parameter count doesn't depend on the sequence length.
2 Update the hidden state.
h = torch.____(W_x @ x_t + W_h @ h + b)h = torch.tanh(W_x @ x_t + W_h @ h + b)Explanation: The classic RNN uses the tanh activation.
3 Match the problems with the forms.
| Term | Meaning |
|---|---|
Sentiment analysis |
Many to one |
Part-of-speech tagging |
Many to many |
Text generation |
One to many |
hidden state |
Summary of the past |