Agent
- Machine Learning course
- Expert
- Lesson 36/44
- 5 min
An agent selects actions by interacting with an environment.
The agent
The agent is the part that takes the observation, chooses an action and learns from experience. The simplest agent behaves randomly; this is the baseline every method must beat:
class RastgeleAjan:
def __init__(self, rng):
self.rng = rng
def sec(self, durum):
return int(self.rng.integers(2))An agent's performance is measured not by a single episode but by the average total reward over many episodes, because the environment or the agent may be random.
The episode limit
A random agent may never reach the goal; so an episode gets a step limit (truncated in Gymnasium). An episode that hits the limit counts as "failed":
def bolum(ajan, sinir=50):
durum, toplam = 0, 0.0
for t in range(sinir):
durum, odul, bitti = adim(durum, ajan.sec(durum))
toplam += odul
if bitti:
return toplam, t + 1
return toplam, sinirThe average episode length is also a useful measure: a good agent crosses the corridor in 4 steps.
Practice Random and fixed agents
Run RastgeleAjan and an always-right SagAjan for 1000 episodes with the given bolum function. For each, print the average total reward and the average episode length with round(..., 2) on one line.
Starter code
import numpy as np
N = 5 # durumlar 0..4, hedef 4
SOL, SAG = 0, 1
def adim(durum, eylem):
yeni = max(0, durum - 1) if eylem == SOL else durum + 1
if yeni == N - 1:
return yeni, 1.0, True
return yeni, -0.1, False
def bolum(ajan, sinir=50):
durum, toplam = 0, 0.0
for t in range(sinir):
durum, odul, bitti = adim(durum, ajan.sec(durum))
toplam += odul
if bitti:
return toplam, t + 1
return toplam, sinir
rng = np.random.default_rng(0)
# buraya yazSample solution
import numpy as np
N = 5 # durumlar 0..4, hedef 4
SOL, SAG = 0, 1
def adim(durum, eylem):
yeni = max(0, durum - 1) if eylem == SOL else durum + 1
if yeni == N - 1:
return yeni, 1.0, True
return yeni, -0.1, False
def bolum(ajan, sinir=50):
durum, toplam = 0, 0.0
for t in range(sinir):
durum, odul, bitti = adim(durum, ajan.sec(durum))
toplam += odul
if bitti:
return toplam, t + 1
return toplam, sinir
rng = np.random.default_rng(0)
class RastgeleAjan:
def sec(self, durum):
return int(rng.integers(2))
class SagAjan:
def sec(self, durum):
return SAG
for ajan in [RastgeleAjan(), SagAjan()]:
s = np.array([bolum(ajan) for _ in range(1000)])
print(round(s[:, 0].mean(), 2), round(s[:, 1].mean(), 2))Expected output
-0.89 19.42 0.7 4.0
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why is an agent's performance measured by the average of many episodes?
- Because the agent or environment may be random, a single episode is misleading Correct answer
- The reward is only given at the end
- The average is always zero
- Episodes depend on each other
Explanation: Many episodes are averaged to reduce variance.
2 Choose a random action.
return int(self.rng.____(2))return int(self.rng.integers(2))Explanation: integers(2) returns 0 or 1.
3 Match.
| Term | Meaning |
|---|---|
random agent |
Baseline |
step limit |
Truncated episode |
mean reward |
Performance measure |
4 steps |
Best corridor time |