Machine Learning
Challenge 4
Define an RL agent policy.
Challenge: near reward or big reward?
In a 7-cell corridor the agent starts at 2. The left end (0) is near and gives 0.5 reward; the right end (6) is far but gives 2.0. The discount factor γ determines which path is chosen:
# left: 2 steps -> value 0.5 · γ
# right: 4 steps -> value 2.0 · γ³
# γ = 0.3: left 0.15 > right 0.054 (impatient)
# γ = 0.9: left 0.45 < right 1.458 (patient)
In this challenge you'll first learn the policy with Q-learning, then find the exact solution in the known environment with value iteration and compare the two γ values.
Exercise Learn with Q-learning
Write an q_tablosu(gamma) function that trains Q-learning in the given environment (alfa=0.5, eps=0.3, 2000 episodes, each from state 2) separately for γ = 0.3 and γ = 0.9. For each γ print the greedy policy of the inner states (1..5) (0 = SOL, 1 = SAG).
Show example solution
import numpy as np
N = 7 # durumlar 0..6; 0 ve 6 bitiş
SOL, SAG = 0, 1
def adim(s, a):
yeni = s - 1 if a == SOL else s + 1
if yeni == 0:
return yeni, 0.5, True
if yeni == N - 1:
return yeni, 2.0, True
return yeni, 0.0, False
rng = np.random.default_rng(0)
def q_tablosu(gamma, alfa=0.5, eps=0.3, bolum=2000):
Q = np.zeros((N, 2))
for _ in range(bolum):
s, bitti = 2, False
while not bitti:
a = int(rng.integers(2)) if rng.random() < eps else int(np.argmax(Q[s]))
y, r, bitti = adim(s, a)
Q[s, a] += alfa * (r + gamma * Q[y].max() * (not bitti) - Q[s, a])
s = y
return Q
for gamma in [0.3, 0.9]:
print(gamma, q_tablosu(gamma)[1:N - 1].argmax(axis=1))
Exercise Value iteration
Since the environment model is known, write a deger_yinelemesi(gamma, tur=100) function: in each round, compute the outcome of both actions for each inner state with adim and update V[s] = max(r + γ V[y] (if not terminal)). For γ = 0.3 and 0.9, print the value of the start state 2 with round(..., 3) and the policy of the inner states on one line.
Show example solution
import numpy as np
N = 7 # durumlar 0..6; 0 ve 6 bitiş
SOL, SAG = 0, 1
def adim(s, a):
yeni = s - 1 if a == SOL else s + 1
if yeni == 0:
return yeni, 0.5, True
if yeni == N - 1:
return yeni, 2.0, True
return yeni, 0.0, False
def deger_yinelemesi(gamma, tur=100):
V = np.zeros(N)
for _ in range(tur):
V_yeni = V.copy()
for s in range(1, N - 1):
sonuclar = [adim(s, a) for a in (SOL, SAG)]
V_yeni[s] = max(r + gamma * V[y] * (not b) for y, r, b in sonuclar)
V = V_yeni
politika = [int(np.argmax([r + gamma * V[y] * (not b) for y, r, b in [adim(s, a) for a in (SOL, SAG)]])) for s in range(1, N - 1)]
return V, politika
for gamma in [0.3, 0.9]:
V, politika = deger_yinelemesi(gamma)
print(gamma, round(V[2], 3), politika)
Test yourself
-
With γ = 0.9, which way should the agent at the start go?
- It should stay in place
- Right (to the 2.0 reward)
- Left (to the 0.5 reward)
- It doesn't matter
Show answer
Right (to the 2.0 reward) — 2.0 · 0.9³ = 1.458 > 0.5 · 0.9 = 0.45.
-
In value iteration, take the value of the best action.
V_yeni[s] = ____(r + gamma * V[y] * (not b) for y, r, b in sonuclar)
Show answer
V_yeni[s] = max(r + gamma * V[y] * (not b) for y, r, b in sonuclar)
-
Match.
- γ = 0.3
- Near reward
- γ = 0.9
- Big reward
- Q-learning
- Model-free learning
- value iteration
- Exact solution when the model is known
Next lesson: Expert section exam
