Q-learning
- Machine Learning course
- Expert
- Lesson 39/44
- 5 min
Q-learning is an RL algorithm that learns state-action values.
Q-learning
Q-learning keeps the value of each (state, action) pair, Q(s, a), in a table and updates it toward the Bellman target after each step:
hedef = odul + gamma * Q[yeni].max() * (not bitti)
Q[durum, eylem] += alfa * (hedef - Q[durum, eylem])The update uses the best action of the next state; it's independent of what the agent actually did (off-policy). Thanks to this it learns the best policy even while exploring ε-greedily.
The training loop
The Q table starts at zero; ε-greedy episodes are played and a single cell is updated at each step. After enough episodes the greedy policy (Q.argmax(axis=1)) converges to the best policy:
Q = np.zeros((N, 2))
for b in range(300):
durum, bitti = 0, False
while not bitti:
eylem = eps_acgozlu(Q, durum, eps=0.2)
yeni, odul, bitti = adim(durum, eylem)
hedef = odul + gamma * Q[yeni].max() * (not bitti)
Q[durum, eylem] += alfa * (hedef - Q[durum, eylem])
durum = yeniIn large state spaces a neural network is used instead of a table: DQN (2015) learned Atari games this way.
Practice Q-learning in the corridor
Train Q-learning with the given environment and eps_acgozlu for 300 episodes with alfa=0.5, gamma=0.9, eps=0.2. Print the greedy policy of the non-goal states (Q[:N-1].argmax(axis=1)), then the SAG column of the Q table with np.round(..., 3) on separate lines.
Starter code
import numpy as np
N = 5 # durumlar 0..4, hedef 4
SOL, SAG = 0, 1
def adim(durum, eylem):
yeni = max(0, durum - 1) if eylem == SOL else durum + 1
if yeni == N - 1:
return yeni, 1.0, True
return yeni, -0.1, False
rng = np.random.default_rng(0)
def eps_acgozlu(Q, durum, eps):
if rng.random() < eps:
return int(rng.integers(2))
return int(np.argmax(Q[durum]))
alfa, gamma = 0.5, 0.9
# buraya yazSample solution
import numpy as np
N = 5 # durumlar 0..4, hedef 4
SOL, SAG = 0, 1
def adim(durum, eylem):
yeni = max(0, durum - 1) if eylem == SOL else durum + 1
if yeni == N - 1:
return yeni, 1.0, True
return yeni, -0.1, False
rng = np.random.default_rng(0)
def eps_acgozlu(Q, durum, eps):
if rng.random() < eps:
return int(rng.integers(2))
return int(np.argmax(Q[durum]))
alfa, gamma = 0.5, 0.9
Q = np.zeros((N, 2))
for b in range(300):
durum, bitti = 0, False
while not bitti:
eylem = eps_acgozlu(Q, durum, 0.2)
yeni, odul, bitti = adim(durum, eylem)
hedef = odul + gamma * Q[yeni].max() * (not bitti)
Q[durum, eylem] += alfa * (hedef - Q[durum, eylem])
durum = yeni
print(Q[:N - 1].argmax(axis=1))
print(np.round(Q[:, SAG], 3))Expected output
[1 1 1 1] [0.458 0.62 0.8 1. 0. ]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why is Q[yeni].max() not used in the target when the episode ends?
- The reward is negative
- There are no future rewards after the terminal state Correct answer
- The Q table is erased
- γ becomes zero
Explanation: The target is just the final reward.
2 Put the next state's best value into the Bellman target.
hedef = odul + gamma * Q[yeni].____() * (not bitti)hedef = odul + gamma * Q[yeni].max() * (not bitti)Explanation: Q-learning uses the value of the best action.
3 Match.
| Term | Meaning |
|---|---|
alfa |
Learning rate |
gamma |
Discount factor |
hedef − Q |
Temporal-difference error |
DQN |
Q-learning with a neural network |