Reward
- Machine Learning course
- Expert
- Lesson 38/44
- 5 min
Reward is a signal showing how good the agent's action was.
Reward and return
The agent tries to maximize not the single-step reward but the return. Future rewards are reduced by a discount factor γ:
# G_t = r_t + γ r_{t+1} + γ² r_{t+2} + ...
G = sum(gamma ** k * r for k, r in enumerate(oduller))If γ = 0 the agent cares only about the immediate reward; as γ → 1 it also takes the distant future into account at almost full value. γ < 1 also keeps the return finite in infinite episodes.
Computing the return backwards
Each step's return can be computed from the next one's: G_t = r_t + γ G_{t+1}. Going from the end of the episode to the start, all returns are found in a single pass:
def getiriler(oduller, gamma):
G, sonuc = 0.0, []
for r in reversed(oduller):
G = r + gamma * G
sonuc.append(G)
return sonuc[::-1]This recursion is the core of the Bellman equation on which Q-learning and all value-based methods rest.
Practice Discounted returns
Write the backward-running getiriler(oduller, gamma) function. For the corridor's best-path rewards [-0.1, -0.1, -0.1, 1.0], print the returns with γ = 0.9, and the first step's return with γ = 0.0, with np.round(..., 3) on separate lines. Then print with np.isclose that the forward formula (sum(gamma ** k * r ...)) gives the same first return.
Starter code
import numpy as np
oduller = [-0.1, -0.1, -0.1, 1.0]
# buraya yazSample solution
import numpy as np
oduller = [-0.1, -0.1, -0.1, 1.0]
def getiriler(oduller, gamma):
G, sonuc = 0.0, []
for r in reversed(oduller):
G = r + gamma * G
sonuc.append(G)
return sonuc[::-1]
print(np.round(getiriler(oduller, 0.9), 3))
print(np.round(getiriler(oduller, 0.0)[0], 3))
ileri = sum(0.9 ** k * r for k, r in enumerate(oduller))
print(np.isclose(ileri, getiriler(oduller, 0.9)[0]))Expected output
[0.458 0.62 0.8 1. ] -0.1 True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 With γ = 0.5 and rewards [0, 0, 4], what is the return at the first step?
- 2
- 0
- 1 Correct answer
- 4
Explanation: 0 + 0.5·0 + 0.25·4 = 1.
2 Compute the return from the next one.
G = r + ____ * GG = r + gamma * GExplanation: G_t = r_t + γ G_{t+1}.
3 Match.
| Term | Meaning |
|---|---|
γ = 0 |
Only immediate reward |
γ → 1 |
Distant future matters |
G_t |
Discounted sum |
Bellman |
G_t = r_t + γ G_{t+1} |