Machine Learning
Reward
Reward is a signal showing how good the agent's action was.
Reward and return
The agent tries to maximize not the single-step reward but the return. Future rewards are reduced by a discount factor γ:
# G_t = r_t + γ r_{t+1} + γ² r_{t+2} + ...
G = sum(gamma ** k * r for k, r in enumerate(oduller))
If γ = 0 the agent cares only about the immediate reward; as γ → 1 it also takes the distant future into account at almost full value. γ < 1 also keeps the return finite in infinite episodes.
Computing the return backwards
Each step's return can be computed from the next one's: G_t = r_t + γ G_{t+1}. Going from the end of the episode to the start, all returns are found in a single pass:
def getiriler(oduller, gamma):
G, sonuc = 0.0, []
for r in reversed(oduller):
G = r + gamma * G
sonuc.append(G)
return sonuc[::-1]
This recursion is the core of the Bellman equation on which Q-learning and all value-based methods rest.
Exercise Discounted returns
Write the backward-running getiriler(oduller, gamma) function. For the corridor's best-path rewards [-0.1, -0.1, -0.1, 1.0], print the returns with γ = 0.9, and the first step's return with γ = 0.0, with np.round(..., 3) on separate lines. Then print with np.isclose that the forward formula (sum(gamma ** k * r ...)) gives the same first return.
Show example solution
import numpy as np
oduller = [-0.1, -0.1, -0.1, 1.0]
def getiriler(oduller, gamma):
G, sonuc = 0.0, []
for r in reversed(oduller):
G = r + gamma * G
sonuc.append(G)
return sonuc[::-1]
print(np.round(getiriler(oduller, 0.9), 3))
print(np.round(getiriler(oduller, 0.0)[0], 3))
ileri = sum(0.9 ** k * r for k, r in enumerate(oduller))
print(np.isclose(ileri, getiriler(oduller, 0.9)[0]))
Test yourself
-
With γ = 0.5 and rewards [0, 0, 4], what is the return at the first step?
Show answer
1 — 0 + 0.5·0 + 0.25·4 = 1.
-
Compute the return from the next one.
G = r + ____ * G
Show answer
G = r + gamma * G
-
Match.
- γ = 0
- Only immediate reward
- γ → 1
- Distant future matters
- G_t
- Discounted sum
- Bellman
- G_t = r_t + γ G_{t+1}
Next lesson: Q-learning
