Gates

LSTM controls information flow with input, forget, and output gates.

The gates

At every step of an LSTM four components are computed; three are gates that answer "how much" between 0 and 1 with sigmoid:

Forget gate f: how much of the old cell information to keep.
Input gate i: how much of the new candidate information to write.
Candidate g (tanh): the new information to write.
Output gate o: how much of the cell to read into the hidden state.

f = sigmoid(W_f [h, x]);  i = sigmoid(W_i [h, x]);  o = sigmoid(W_o [h, x])
g = tanh(W_g [h, x])
c = f * c + i * g
h = o * tanh(c)

One step by hand

When the gate values are given, a step is easy to compute:

c_eski = torch.tensor([2.0])
f, i, o = torch.tensor([0.5]), torch.tensor([1.0]), torch.tensor([1.0])
g = torch.tensor([0.5])
c = f * c_eski + i * g       # 0.5·2 + 1·0.5 = 1.5
h = o * torch.tanh(c)        # tanh(1.5) ≈ 0.905

In a real LSTM the gates aren't fixed; they are computed at every step from the input and the previous hidden state.

Exercise An LSTM step

Compute an LSTM step with the given gate values: c = f * c_eski + i * g, h = o * torch.tanh(c). Print c and h with round(..., 3) on one line. Then set f = 0, repeat the computation and print the new c.

Show example solution

import torch

c_eski = torch.tensor(2.0)
i = torch.tensor(1.0)
o = torch.tensor(0.5)
g = torch.tensor(0.5)
f = torch.tensor(0.5)

c = f * c_eski + i * g
h = o * torch.tanh(c)
print(round(c.item(), 3), round(h.item(), 3))
f = torch.tensor(0.0)
print((f * c_eski + i * g).item())

Test yourself

What happens if the forget gate is close to 0?
- No new information is written
- The output becomes zero
- The old cell information is erased
- Everything is kept
Show answer

The old cell information is erased — c = f · c_old + ...; f ≈ 0 wipes out the old information by multiplication.

Update the cell state.

c = f * c_eski ____ i * g

Show answer

c = f * c_eski + i * g

Match the gates.

f

Forget
i

Input
g

Candidate information
o

Output

Next lesson: Cell state

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies