Policy
- Machine Learning course
- Expert
- Lesson 37/44
- 5 min
Policy is the agent's strategy for selecting actions based on state.
The policy
The policy (π) is the agent's rule of behaviour: it says which action to choose in each state.
- Deterministic: π(s) = a, e.g. a table.
- Stochastic: π(a | s) is a probability distribution; policy-gradient methods learn this.
politika = np.array([SAG, SAG, SAG, SAG, SAG]) # table: state -> action
eylem = politika[durum]The goal of RL is to find the policy that maximizes the expected total reward: π* = argmax_π E[G].
The ε-greedy policy
The most common way to provide exploration is the ε-greedy policy: with probability ε a random action is chosen, otherwise the best known action. As training progresses, ε is usually decreased:
def eps_acgozlu(Q, durum, eps):
if rng.random() < eps:
return int(rng.integers(Q.shape[1]))
return int(np.argmax(Q[durum]))With 2 actions and ε = 0.2, the best action is chosen with probability 0.8 + 0.2/2 = 0.9, since the random choice also sometimes finds the best action.
Practice ε-greedy selection
Write the eps_acgozlu(Q, durum, eps) function. In the given Q table, make 10 000 choices for state 0 with ε = 0, 0.2 and 1.0; print the share of choices of the best action with round(..., 2) on one line. Then print the greedy policy from the Q table (each state's argmax).
Starter code
import numpy as np
rng = np.random.default_rng(0)
Q = np.array([[0.1, 0.5], [0.3, 0.6], [0.7, 0.2], [0.0, 0.9]])
# buraya yazSample solution
import numpy as np
rng = np.random.default_rng(0)
Q = np.array([[0.1, 0.5], [0.3, 0.6], [0.7, 0.2], [0.0, 0.9]])
def eps_acgozlu(Q, durum, eps):
if rng.random() < eps:
return int(rng.integers(Q.shape[1]))
return int(np.argmax(Q[durum]))
oranlar = []
for eps in [0, 0.2, 1.0]:
secim = [eps_acgozlu(Q, 0, eps) for _ in range(10_000)]
oranlar.append(round(np.mean(np.array(secim) == 1), 2))
print(*oranlar)
print(Q.argmax(axis=1))Expected output
1.0 0.9 0.5 [1 1 0 1]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What is the exploration–exploitation dilemma?
- The split between training and test data
- Choosing the learning rate
- Normalizing the reward
- The balance between choosing the best known action and trying new ones Correct answer
Explanation: An agent that always picks the known one never discovers anything better.
2 Choose the best known action.
return int(np.____(Q[durum]))return int(np.argmax(Q[durum]))Explanation: argmax gives the action with the highest Q value.
3 Match.
| Term | Meaning |
|---|---|
π(s) = a |
Deterministic policy |
π(a | s) |
Stochastic policy |
ε |
Exploration probability |
argmax Q |
Exploitation |