Challenge 4
- Machine Learning course
- Expert
- Lesson 44/44
- 8 min
Define an RL agent policy.
Challenge: near reward or big reward?
In a 7-cell corridor the agent starts at 2. The left end (0) is near and gives 0.5 reward; the right end (6) is far but gives 2.0. The discount factor γ determines which path is chosen:
# left: 2 steps -> value 0.5 · γ
# right: 4 steps -> value 2.0 · γ³
# γ = 0.3: left 0.15 > right 0.054 (impatient)
# γ = 0.9: left 0.45 < right 1.458 (patient)In this challenge you'll first learn the policy with Q-learning, then find the exact solution in the known environment with value iteration and compare the two γ values.
Practice Learn with Q-learning
Write an q_tablosu(gamma) function that trains Q-learning in the given environment (alfa=0.5, eps=0.3, 2000 episodes, each from state 2) separately for γ = 0.3 and γ = 0.9. For each γ print the greedy policy of the inner states (1..5) (0 = SOL, 1 = SAG).
Starter code
import numpy as np
N = 7 # durumlar 0..6; 0 ve 6 bitiş
SOL, SAG = 0, 1
def adim(s, a):
yeni = s - 1 if a == SOL else s + 1
if yeni == 0:
return yeni, 0.5, True
if yeni == N - 1:
return yeni, 2.0, True
return yeni, 0.0, False
rng = np.random.default_rng(0)
# buraya yazSample solution
import numpy as np
N = 7 # durumlar 0..6; 0 ve 6 bitiş
SOL, SAG = 0, 1
def adim(s, a):
yeni = s - 1 if a == SOL else s + 1
if yeni == 0:
return yeni, 0.5, True
if yeni == N - 1:
return yeni, 2.0, True
return yeni, 0.0, False
rng = np.random.default_rng(0)
def q_tablosu(gamma, alfa=0.5, eps=0.3, bolum=2000):
Q = np.zeros((N, 2))
for _ in range(bolum):
s, bitti = 2, False
while not bitti:
a = int(rng.integers(2)) if rng.random() < eps else int(np.argmax(Q[s]))
y, r, bitti = adim(s, a)
Q[s, a] += alfa * (r + gamma * Q[y].max() * (not bitti) - Q[s, a])
s = y
return Q
for gamma in [0.3, 0.9]:
print(gamma, q_tablosu(gamma)[1:N - 1].argmax(axis=1))Expected output
0.3 [0 0 1 1 1] 0.9 [1 1 1 1 1]
Practice Value iteration
Since the environment model is known, write a deger_yinelemesi(gamma, tur=100) function: in each round, compute the outcome of both actions for each inner state with adim and update V[s] = max(r + γ V[y] (if not terminal)). For γ = 0.3 and 0.9, print the value of the start state 2 with round(..., 3) and the policy of the inner states on one line.
Starter code
import numpy as np
N = 7 # durumlar 0..6; 0 ve 6 bitiş
SOL, SAG = 0, 1
def adim(s, a):
yeni = s - 1 if a == SOL else s + 1
if yeni == 0:
return yeni, 0.5, True
if yeni == N - 1:
return yeni, 2.0, True
return yeni, 0.0, False
# buraya yazSample solution
import numpy as np
N = 7 # durumlar 0..6; 0 ve 6 bitiş
SOL, SAG = 0, 1
def adim(s, a):
yeni = s - 1 if a == SOL else s + 1
if yeni == 0:
return yeni, 0.5, True
if yeni == N - 1:
return yeni, 2.0, True
return yeni, 0.0, False
def deger_yinelemesi(gamma, tur=100):
V = np.zeros(N)
for _ in range(tur):
V_yeni = V.copy()
for s in range(1, N - 1):
sonuclar = [adim(s, a) for a in (SOL, SAG)]
V_yeni[s] = max(r + gamma * V[y] * (not b) for y, r, b in sonuclar)
V = V_yeni
politika = [int(np.argmax([r + gamma * V[y] * (not b) for y, r, b in [adim(s, a) for a in (SOL, SAG)]])) for s in range(1, N - 1)]
return V, politika
for gamma in [0.3, 0.9]:
V, politika = deger_yinelemesi(gamma)
print(gamma, round(V[2], 3), politika)Expected output
0.3 0.15 [0, 0, 1, 1, 1] 0.9 1.458 [1, 1, 1, 1, 1]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 With γ = 0.9, which way should the agent at the start go?
- It should stay in place
- Right (to the 2.0 reward) Correct answer
- Left (to the 0.5 reward)
- It doesn't matter
Explanation: 2.0 · 0.9³ = 1.458 > 0.5 · 0.9 = 0.45.
2 In value iteration, take the value of the best action.
V_yeni[s] = ____(r + gamma * V[y] * (not b) for y, r, b in sonuclar)V_yeni[s] = max(r + gamma * V[y] * (not b) for y, r, b in sonuclar)Explanation: The Bellman optimality equation: V(s) = max_a [r + γ V(s')].
3 Match.
| Term | Meaning |
|---|---|
γ = 0.3 |
Near reward |
γ = 0.9 |
Big reward |
Q-learning |
Model-free learning |
value iteration |
Exact solution when the model is known |