Summary - RL
- Machine Learning course
- Expert
- Lesson 40/44
- 6 min
In this group you learned agent, policy, reward, and Q-learning.
Summary: reinforcement learning
In this group:
- RL: the agent interacts with the environment and learns from reward; the goal is the total (discounted) return.
- Agents are compared by the average over many episodes; the random agent is the baseline.
- The policy is the rule from state to action; ε-greedy balances exploration and exploitation.
- The return is computed backwards with
G_t = r_t + γ G_{t+1}(Bellman). - Q-learning updates the Q table toward the target
r + γ max Q(s'); it is off-policy.
# Value-based: Q-learning, DQN
# Policy-based: REINFORCE, PPO (used in RLHF)
# Actor-critic: A2C, SACPractice Evaluate the learned policy
Run the given trained Q-learning loop. Then write a degerlendir(Q) function that plays an episode with the exploration-free (greedy) policy; print the path and the total reward (round(..., 2)) on one line. Finally, to show that with an untrained (zero) Q table the same evaluation gets stuck at the 20-step limit, print the path's length.
Starter code
import numpy as np
N = 5 # durumlar 0..4, hedef 4
SOL, SAG = 0, 1
def adim(durum, eylem):
yeni = max(0, durum - 1) if eylem == SOL else durum + 1
if yeni == N - 1:
return yeni, 1.0, True
return yeni, -0.1, False
rng = np.random.default_rng(0)
Q = np.zeros((N, 2))
for b in range(300):
durum, bitti = 0, False
while not bitti:
eylem = int(rng.integers(2)) if rng.random() < 0.2 else int(np.argmax(Q[durum]))
yeni, odul, bitti = adim(durum, eylem)
Q[durum, eylem] += 0.5 * (odul + 0.9 * Q[yeni].max() * (not bitti) - Q[durum, eylem])
durum = yeni
# buraya yazSample solution
import numpy as np
N = 5 # durumlar 0..4, hedef 4
SOL, SAG = 0, 1
def adim(durum, eylem):
yeni = max(0, durum - 1) if eylem == SOL else durum + 1
if yeni == N - 1:
return yeni, 1.0, True
return yeni, -0.1, False
rng = np.random.default_rng(0)
Q = np.zeros((N, 2))
for b in range(300):
durum, bitti = 0, False
while not bitti:
eylem = int(rng.integers(2)) if rng.random() < 0.2 else int(np.argmax(Q[durum]))
yeni, odul, bitti = adim(durum, eylem)
Q[durum, eylem] += 0.5 * (odul + 0.9 * Q[yeni].max() * (not bitti) - Q[durum, eylem])
durum = yeni
def degerlendir(Q, sinir=20):
durum, toplam, yol = 0, 0.0, [0]
for _ in range(sinir):
durum, odul, bitti = adim(durum, int(np.argmax(Q[durum])))
toplam += odul
yol.append(durum)
if bitti:
break
return yol, toplam
yol, toplam = degerlendir(Q)
print(yol, round(toplam, 2))
print(len(degerlendir(np.zeros((N, 2)))[0]))Expected output
[0, 1, 2, 3, 4] 0.7 21
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 How is a policy obtained from a learned Q table?
- The action with the highest Q value is chosen in each state Correct answer
- The Q values are summed
- A random action is chosen
- The lowest Q is chosen
Explanation: Greedy policy: π(s) = argmax_a Q(s, a).
2 Extract the greedy policy from the Q table.
Question
politika = Q.____(axis=1)
Answer
politika = Q.argmax(axis=1)Explanation: The best action of each row (state).
3 Match.
| Term | Meaning |
|---|---|
Q-learning |
Value-based |
PPO |
Policy-based |
ε |
Exploration |
γ |
Discount |