Summary - RL

In this group you learned agent, policy, reward, and Q-learning.

Summary: reinforcement learning

In this group:

RL: the agent interacts with the environment and learns from reward; the goal is the total (discounted) return.
Agents are compared by the average over many episodes; the random agent is the baseline.
The policy is the rule from state to action; ε-greedy balances exploration and exploitation.
The return is computed backwards with G_t = r_t + γ G_{t+1} (Bellman).
Q-learning updates the Q table toward the target r + γ max Q(s'); it is off-policy.

# Value-based: Q-learning, DQN
# Policy-based: REINFORCE, PPO (used in RLHF)
# Actor-critic: A2C, SAC

Exercise Evaluate the learned policy

Run the given trained Q-learning loop. Then write a degerlendir(Q) function that plays an episode with the exploration-free (greedy) policy; print the path and the total reward (round(..., 2)) on one line. Finally, to show that with an untrained (zero) Q table the same evaluation gets stuck at the 20-step limit, print the path's length.

Show example solution

import numpy as np

N = 5            # durumlar 0..4, hedef 4
SOL, SAG = 0, 1

def adim(durum, eylem):
    yeni = max(0, durum - 1) if eylem == SOL else durum + 1
    if yeni == N - 1:
        return yeni, 1.0, True
    return yeni, -0.1, False

rng = np.random.default_rng(0)
Q = np.zeros((N, 2))
for b in range(300):
    durum, bitti = 0, False
    while not bitti:
        eylem = int(rng.integers(2)) if rng.random() < 0.2 else int(np.argmax(Q[durum]))
        yeni, odul, bitti = adim(durum, eylem)
        Q[durum, eylem] += 0.5 * (odul + 0.9 * Q[yeni].max() * (not bitti) - Q[durum, eylem])
        durum = yeni

def degerlendir(Q, sinir=20):
    durum, toplam, yol = 0, 0.0, [0]
    for _ in range(sinir):
        durum, odul, bitti = adim(durum, int(np.argmax(Q[durum])))
        toplam += odul
        yol.append(durum)
        if bitti:
            break
    return yol, toplam

yol, toplam = degerlendir(Q)
print(yol, round(toplam, 2))
print(len(degerlendir(np.zeros((N, 2)))[0]))

Test yourself

How is a policy obtained from a learned Q table?
- The action with the highest Q value is chosen in each state
- The Q values are summed
- A random action is chosen
- The lowest Q is chosen
Show answer

The action with the highest Q value is chosen in each state — Greedy policy: π(s) = argmax_a Q(s, a).
Extract the greedy policy from the Q table.
```
politika = Q.____(axis=1)
```
Show answer
```
politika = Q.argmax(axis=1)
```
Match.

Q-learning

Value-based
PPO

Policy-based
ε

Exploration
γ

Discount

Next lesson: Challenge 1

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies