Machine Learning
Agent
An agent selects actions by interacting with an environment.
The agent
The agent is the part that takes the observation, chooses an action and learns from experience. The simplest agent behaves randomly; this is the baseline every method must beat:
class RastgeleAjan:
def __init__(self, rng):
self.rng = rng
def sec(self, durum):
return int(self.rng.integers(2))
An agent's performance is measured not by a single episode but by the average total reward over many episodes, because the environment or the agent may be random.
The episode limit
A random agent may never reach the goal; so an episode gets a step limit (truncated in Gymnasium). An episode that hits the limit counts as "failed":
def bolum(ajan, sinir=50):
durum, toplam = 0, 0.0
for t in range(sinir):
durum, odul, bitti = adim(durum, ajan.sec(durum))
toplam += odul
if bitti:
return toplam, t + 1
return toplam, sinir
The average episode length is also a useful measure: a good agent crosses the corridor in 4 steps.
Exercise Random and fixed agents
Run RastgeleAjan and an always-right SagAjan for 1000 episodes with the given bolum function. For each, print the average total reward and the average episode length with round(..., 2) on one line.
Show example solution
import numpy as np
N = 5 # durumlar 0..4, hedef 4
SOL, SAG = 0, 1
def adim(durum, eylem):
yeni = max(0, durum - 1) if eylem == SOL else durum + 1
if yeni == N - 1:
return yeni, 1.0, True
return yeni, -0.1, False
def bolum(ajan, sinir=50):
durum, toplam = 0, 0.0
for t in range(sinir):
durum, odul, bitti = adim(durum, ajan.sec(durum))
toplam += odul
if bitti:
return toplam, t + 1
return toplam, sinir
rng = np.random.default_rng(0)
class RastgeleAjan:
def sec(self, durum):
return int(rng.integers(2))
class SagAjan:
def sec(self, durum):
return SAG
for ajan in [RastgeleAjan(), SagAjan()]:
s = np.array([bolum(ajan) for _ in range(1000)])
print(round(s[:, 0].mean(), 2), round(s[:, 1].mean(), 2))
Test yourself
-
Why is an agent's performance measured by the average of many episodes?
- Because the agent or environment may be random, a single episode is misleading
- The reward is only given at the end
- The average is always zero
- Episodes depend on each other
Show answer
Because the agent or environment may be random, a single episode is misleading — Many episodes are averaged to reduce variance.
-
Choose a random action.
return int(self.rng.____(2))
Show answer
return int(self.rng.integers(2))
-
Match.
- random agent
- Baseline
- step limit
- Truncated episode
- mean reward
- Performance measure
- 4 steps
- Best corridor time
Next lesson: Policy
