Gates
- Machine Learning course
- Advanced
- Lesson 54/78
- 5 min
LSTM controls information flow with input, forget, and output gates.
The gates
At every step of an LSTM four components are computed; three are gates that answer "how much" between 0 and 1 with sigmoid:
- Forget gate f: how much of the old cell information to keep.
- Input gate i: how much of the new candidate information to write.
- Candidate g (tanh): the new information to write.
- Output gate o: how much of the cell to read into the hidden state.
f = sigmoid(W_f [h, x]); i = sigmoid(W_i [h, x]); o = sigmoid(W_o [h, x])
g = tanh(W_g [h, x])
c = f * c + i * g
h = o * tanh(c)One step by hand
When the gate values are given, a step is easy to compute:
c_eski = torch.tensor([2.0])
f, i, o = torch.tensor([0.5]), torch.tensor([1.0]), torch.tensor([1.0])
g = torch.tensor([0.5])
c = f * c_eski + i * g # 0.5·2 + 1·0.5 = 1.5
h = o * torch.tanh(c) # tanh(1.5) ≈ 0.905In a real LSTM the gates aren't fixed; they are computed at every step from the input and the previous hidden state.
Practice An LSTM step
Compute an LSTM step with the given gate values: c = f * c_eski + i * g, h = o * torch.tanh(c). Print c and h with round(..., 3) on one line. Then set f = 0, repeat the computation and print the new c.
Starter code
import torch
c_eski = torch.tensor(2.0)
i = torch.tensor(1.0)
o = torch.tensor(0.5)
g = torch.tensor(0.5)
f = torch.tensor(0.5)
# buraya yazSample solution
import torch
c_eski = torch.tensor(2.0)
i = torch.tensor(1.0)
o = torch.tensor(0.5)
g = torch.tensor(0.5)
f = torch.tensor(0.5)
c = f * c_eski + i * g
h = o * torch.tanh(c)
print(round(c.item(), 3), round(h.item(), 3))
f = torch.tensor(0.0)
print((f * c_eski + i * g).item())Expected output
1.5 0.453 0.5
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What happens if the forget gate is close to 0?
- No new information is written
- The output becomes zero
- The old cell information is erased Correct answer
- Everything is kept
Explanation: c = f · c_old + ...; f ≈ 0 wipes out the old information by multiplication.
2 Update the cell state.
c = f * c_eski ____ i * gc = f * c_eski + i * gExplanation: The cell is updated additively: kept + written.
3 Match the gates.
| Term | Meaning |
|---|---|
f |
Forget |
i |
Input |
g |
Candidate information |
o |
Output |