Machine Learning
Policy
Policy is the agent's strategy for selecting actions based on state.
The policy
The policy (π) is the agent's rule of behaviour: it says which action to choose in each state.
- Deterministic: π(s) = a, e.g. a table.
- Stochastic: π(a | s) is a probability distribution; policy-gradient methods learn this.
politika = np.array([SAG, SAG, SAG, SAG, SAG]) # table: state -> action
eylem = politika[durum]
The goal of RL is to find the policy that maximizes the expected total reward: π* = argmax_π E[G].
The ε-greedy policy
The most common way to provide exploration is the ε-greedy policy: with probability ε a random action is chosen, otherwise the best known action. As training progresses, ε is usually decreased:
def eps_acgozlu(Q, durum, eps):
if rng.random() < eps:
return int(rng.integers(Q.shape[1]))
return int(np.argmax(Q[durum]))
With 2 actions and ε = 0.2, the best action is chosen with probability 0.8 + 0.2/2 = 0.9, since the random choice also sometimes finds the best action.
Exercise ε-greedy selection
Write the eps_acgozlu(Q, durum, eps) function. In the given Q table, make 10 000 choices for state 0 with ε = 0, 0.2 and 1.0; print the share of choices of the best action with round(..., 2) on one line. Then print the greedy policy from the Q table (each state's argmax).
Show example solution
import numpy as np
rng = np.random.default_rng(0)
Q = np.array([[0.1, 0.5], [0.3, 0.6], [0.7, 0.2], [0.0, 0.9]])
def eps_acgozlu(Q, durum, eps):
if rng.random() < eps:
return int(rng.integers(Q.shape[1]))
return int(np.argmax(Q[durum]))
oranlar = []
for eps in [0, 0.2, 1.0]:
secim = [eps_acgozlu(Q, 0, eps) for _ in range(10_000)]
oranlar.append(round(np.mean(np.array(secim) == 1), 2))
print(*oranlar)
print(Q.argmax(axis=1))
Test yourself
-
What is the exploration–exploitation dilemma?
- The split between training and test data
- Choosing the learning rate
- Normalizing the reward
- The balance between choosing the best known action and trying new ones
Show answer
The balance between choosing the best known action and trying new ones — An agent that always picks the known one never discovers anything better.
-
Choose the best known action.
return int(np.____(Q[durum]))
Show answer
return int(np.argmax(Q[durum]))
-
Match.
- π(s) = a
- Deterministic policy
- π(a | s)
- Stochastic policy
- ε
- Exploration probability
- argmax Q
- Exploitation
Next lesson: Reward
