Machine Learning
Q-learning
Q-learning is an RL algorithm that learns state-action values.
Q-learning
Q-learning keeps the value of each (state, action) pair, Q(s, a), in a table and updates it toward the Bellman target after each step:
hedef = odul + gamma * Q[yeni].max() * (not bitti)
Q[durum, eylem] += alfa * (hedef - Q[durum, eylem])
The update uses the best action of the next state; it's independent of what the agent actually did (off-policy). Thanks to this it learns the best policy even while exploring ε-greedily.
The training loop
The Q table starts at zero; ε-greedy episodes are played and a single cell is updated at each step. After enough episodes the greedy policy (Q.argmax(axis=1)) converges to the best policy:
Q = np.zeros((N, 2))
for b in range(300):
durum, bitti = 0, False
while not bitti:
eylem = eps_acgozlu(Q, durum, eps=0.2)
yeni, odul, bitti = adim(durum, eylem)
hedef = odul + gamma * Q[yeni].max() * (not bitti)
Q[durum, eylem] += alfa * (hedef - Q[durum, eylem])
durum = yeni
In large state spaces a neural network is used instead of a table: DQN (2015) learned Atari games this way.
Exercise Q-learning in the corridor
Train Q-learning with the given environment and eps_acgozlu for 300 episodes with alfa=0.5, gamma=0.9, eps=0.2. Print the greedy policy of the non-goal states (Q[:N-1].argmax(axis=1)), then the SAG column of the Q table with np.round(..., 3) on separate lines.
Show example solution
import numpy as np
N = 5 # durumlar 0..4, hedef 4
SOL, SAG = 0, 1
def adim(durum, eylem):
yeni = max(0, durum - 1) if eylem == SOL else durum + 1
if yeni == N - 1:
return yeni, 1.0, True
return yeni, -0.1, False
rng = np.random.default_rng(0)
def eps_acgozlu(Q, durum, eps):
if rng.random() < eps:
return int(rng.integers(2))
return int(np.argmax(Q[durum]))
alfa, gamma = 0.5, 0.9
Q = np.zeros((N, 2))
for b in range(300):
durum, bitti = 0, False
while not bitti:
eylem = eps_acgozlu(Q, durum, 0.2)
yeni, odul, bitti = adim(durum, eylem)
hedef = odul + gamma * Q[yeni].max() * (not bitti)
Q[durum, eylem] += alfa * (hedef - Q[durum, eylem])
durum = yeni
print(Q[:N - 1].argmax(axis=1))
print(np.round(Q[:, SAG], 3))
Test yourself
-
Why is Q[yeni].max() not used in the target when the episode ends?
- The reward is negative
- There are no future rewards after the terminal state
- The Q table is erased
- γ becomes zero
Show answer
There are no future rewards after the terminal state — The target is just the final reward.
-
Put the next state's best value into the Bellman target.
hedef = odul + gamma * Q[yeni].____() * (not bitti)
Show answer
hedef = odul + gamma * Q[yeni].max() * (not bitti)
-
Match.
- alfa
- Learning rate
- gamma
- Discount factor
- hedef − Q
- Temporal-difference error
- DQN
- Q-learning with a neural network
Next lesson: Summary - RL
