Model Free Control

Lecture 5: Model Free Control

All lectures are building up to this point, to optimize a problem where we do not have access to the underlying MDP. For such problems, we either do not know the underlying MDP, or it is too big to use (e.g. game of Go).

On policy vs Off policy

• On policy learning is to learn on the job. Learn about policy $\pi$ based on experience sampled from $\pi$
• Off policy learning is to learn by observing others. Learn about policy $\pi$ by sampling another robot (or human) experience $\mu$

Start with the simpler case, which is on policy learning. The basic framework is generalized policy iteration (recap), which alternates between:

• Policy evaluation: estimate $v_{\pi }$
• Policy improvement: generate a new ${\pi }^{\prime }\ge \pi$

Naive case: policy iteration with Monte-Carlo evaluation. Basically, we use MC policy evaluation to update our value function, and then do greedy policy improvement. Would this work?

• No. The main problem is that previously when we had access to the underlying MDP, we could do greedy policy improvement because we had access to the transition dynamics. Specifically, when we do policy improvement, we want to compute: $${\pi }^{\prime }(s)=\underset{a}{\mathrm{argmax}}{\mathcal{R}}_s^a+P_{s{s}^{\prime }}^{a}V(s^{\prime })$$
• However in model-free control, we do not have access to $P_{s{s}^{\prime }}^a$, meaning that we do not know what probabilities determine the state we will end up in given action $a$. So there is no clear way to do greedy policy improvement if we only have an estimate of $V(s)\forall s$.
• To deal with this issue, we can do greedy policy improvement over $Q(s,a)$ instead. Then we can simply take: $${\pi }^{\prime }(s)=\underset{a\in \mathcal{A}}{\mathrm{argmax}}Q(s,a)$$

So now we do generalized policy iteration with action-value function.

• Start with $Q,\pi$
• Update action-value function $Q=q_{\pi }$
• Greedily update policy to $\pi =greedy(Q)$

However, we still have another problem, which is the exploration issue. If we act greedily all the time, there is no guarantee that we will explore all states and thus find the optimal policy.

Toy Example: Greedy Action Selection

Choose between two doors:

• Open left door: reward 0. $V(left)=0$
• Open right door: reward +1. $V(right)=+1$
• Open right door: reward +3. $V(right)=+2$
• Open right door: reward +2. $V(right)=+2$

The greedy policy will lock us onto right door forever. But we will never know if the left door actually has higher mean return.

$ϵ$-Greedy Exploration

The simplest idea for ensuring continual exploration.

• Try all $m$ actions with non-zero probability
• With probability $1-ϵ$ choose the greedy action
• With probability $ϵ$ choose an action at random

$$\pi (a|s)=\{\begin{array}{ll}ϵ/m+1-ϵ& \text{ if }{a}^{\ast }={\mathrm{argmax}}_{a\in \mathcal{A}}Q(s,a)\\ ϵ/m& \text{otherwise}\end{array}$$

Note that $ϵ/m$ is added for the first case as well since the action chosen at random can include the greedy policy $a^{\ast }$ as well.

$ϵ$-greedy policy is important because there is a theorem to assure us that we will indeed get a policy improvement on every step.

Theorem. For any $ϵ$-greedy policy $\pi$, the $ϵ$-greedy policy ${\pi }^{\prime }$ with respect to $q_{\pi }$ is an improvement, i.e. $v_{{\pi }^{\prime }}(s)\ge v_{\pi }(s)$

Proof. $$\begin{array}{rl}{q}_{\pi }(s,{\pi }^{\prime }(s))& =\sum _{a\in \mathcal{A}}{\pi }^{\prime }(a|s)q_{\pi }(s,a)\\ & =ϵ/m\sum _{a\in \mathcal{A}}{q}_{\pi }(s,a)+(1-ϵ)\underset{a\in \mathcal{A}}{\max }{q}_{\pi }(s,a)\\ & \ge ϵ/m\sum _{a\in \mathcal{A}}{q}_{\pi }(s,a)+(1-ϵ)\sum _{a\in \mathcal{A}}\frac{\pi (a|s)-ϵ/m}{1-ϵ}{q}_{\pi }(s,a)\\ & =\sum _{a\in \mathcal{A}}\pi (a|s)q_{\pi }(s,a)=v_{\pi }(s)\end{array}$$ Therefore from the policy improvement theorem, $v_{{\pi }^{\prime }}(s)\ge v_{\pi }(s)$.

The key step in the proof is the transition from line 2 to line 3. The idea is that the maximum q-value (by choosing the greedy action) will be greater than or equal to any weighted average of $q_{\pi }(s,a)$. Hence we choose a clever weighted average such that we can end up with ${\sum }_{a\in \mathcal{A}}\pi (a|s)q_{\pi }(s,a)$ in line 4.

Note that it is indeed a weighted average because of the following. Note that $\pi (a|s)$ must sum to 1 over all actions as it is a valid policy. And since there are $m$ unique actions, we multiply the constant $ϵ/m$ by $m$. $$\begin{array}{rl}\sum _{a\in \mathcal{A}}\frac{\pi (a|s)-ϵ/m}{1-ϵ}& =\frac{1}{1-ϵ}\sum _{a\in \mathcal{A}}\left[\pi (a|s)-ϵ/m\right]\\ & =\frac{1}{1-ϵ}\left[1-m\cdot ϵ/m\right]\\ & =1\end{array}$$

An idea that we encountered earlier. We do not need to fully evaluate the policy before we do a greedy improvement. In the context of Monte Carlo policy evaluation, in the extreme case, we can update the policy after every episode instead of gathering many episodes.

How can we guarantee that we find the optimal policy ${\pi }^{\ast }$? We need to ensure that our algorithm balances two things: (i) suitably explore all options and (ii) ensure that at the end, we converge on a greedy policy.

This leads us to GLIE, which is a property that we want our algo to have.

Definition Greedy in the Limit with Infinite Exploration (GLIE).

• All state-action pairs are explored infinitely many times, i.e. $$\underset{k\to \infty }{\lim }{N}_k(s,a)=\infty$$
• The policy converges on a greedy policy, i.e. $$\underset{k\to \infty }{\lim }{\pi }_k(a|s)=1\left(a=\underset{a^{\prime }\in \mathcal{A}}{\mathrm{argmax}}{Q}_k(s,a^{\prime })\right)$$

One simple way to get GLIE is to use $ϵ$-greedy with a decaying schedule for ${ϵ}_k=\frac{1}{k}$.

GLIE Monte Carlo Control

This brings us to GLIE Monte Carlo control.

Algorithm GLIE Monte-Carlo Control.

• Sample kth episode using policy $\pi :\{S_1,A_1,R_2,...,S_T\}\sim \pi$
• For each state $S_t$ and action $A_t$ in the episode, update $$\begin{array}{rl}N(S_t,A_t)& \leftarrow N(S_t,A_t)+1\\ Q(S_t,A_t)& \leftarrow Q(S_t,A_t)+\frac{1}{N(S_t,A_t)}(G_t-Q(S_t,A_t))\end{array}$$
• Improve policy based on the new action-value function: $$\begin{array}{rl}ϵ& \leftarrow 1/k\\ \pi & \leftarrow ϵ\text{-greedy(Q)}\end{array}$$

MC vs TD Control

• TD learning has several advantages over MC:
  • Lower variance
  • Online
  • Can deal with incomplete sequences
• Natural idea: use TD instead of MC in our control loop
  • Apply TD to Q(S, A)
  • Use $ϵ$-greedy policy improvement
  • Update every time step
  • This is probably the most well known RL algorithm (Sarsa)

Sarsa policy evaluation update step: $$Q(S,A)\leftarrow Q(S,A)+\alpha (R+\gamma Q(S^{\prime },A^{\prime })-Q(S,A))$$

Note that we are updating the Q value for one single state-action pair. We take action $A$ on state $S$ and observe reward $R$, and use that to update the Q-value. In addition, we also sample a next action $A^{\prime }$ and corresponding resultant state $S^{\prime }$, and we bootstrap the Q-value to use $Q(S^{\prime },A^{\prime })$ to also update the Q-value. So it corresponds to a one-step lookahead in TD.

So the off-policy control with Sarsa algo. For every time step:

• Policy evaluation with Sarsa: $Q=q_{\pi }$
• Policy improvement using $ϵ$-greedy

Algorithm. Sarsa algorithm for on-policy control.

• Initialize $Q(s,a),\forall s\in \mathcal{S},a\in \mathcal{A}(s)$ arbitrarily
• Repeat (for each episode):
  • Initialize $S$
  • Choose $A$ from $S$ using policy derived from $Q$ (e.g. $ϵ$-greedy choice)
  • Repeat (for each step of episode):
    • Take action $A$, observe $R,S^{\prime }$
    • Choose $A^{\prime }$ from $S^{\prime }$ using policy derived from $Q$ (e.g. $ϵ$-greedy)
    • Update $Q(S,A)\leftarrow Q(S,A)+\alpha [R+\gamma Q(S^{\prime },A^{\prime })-Q(S,A)]$
    • $S\leftarrow S^{\prime }$, $A\leftarrow A^{\prime }$
  • Until $S$ is terminal

Note that this is a fundamentally on-policy algorithm, because the $A^{\prime },S^{\prime }$ that we sample and use to bootstrap is also the next action and state we end up in.

Algorithm. Sarsa converges to the optimal action value function, $Q(s,a)\to q\ast (s,a)$ under the following conditions:

• GLIE sequence of policies ${\pi }_t(a|s)$
• Robbins Monro sequence of step sizes ${\alpha }_t$: $$\begin{array}{rl}\sum _{t=1}^{\infty }{\alpha }_t& =\infty \\ \sum _{t=1}^{\infty }{\alpha }_t^2& <\infty \end{array}$$

$n$-step Sarsa

As before, we saw that $n$-step algorithm gets the best of both worlds in betwen MC and TD. So we do the same here.

Consider the following $n$-step returns for $n=1,2,\infty$:

• $n=1$, $q_t^{(1)}=R_{t+1}+\gamma Q(S_t+1)$
• $n=2$, $q_t^{(2)}=R_{t+1}+\gamma R_{t+2}+{\gamma }^{2}Q(S_t+2)$
• $n=\infty$, $q_t^{(\infty )}=R_{t+1}+\gamma R_{t+2}+...+{\gamma }^{T-1}{R}_t$

Define the $n$-step Q-return: $$q_t^{(n)}=R_{t+1}+\gamma R_{t+2}+...+{\gamma }^{n-1}{R}_{t+n}+{\gamma }^{n}Q(S_{t+n})$$

Sarsa update $Q(s,a)$ towards the n-step Q-return: $$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha \left(q_t^{(n)}-Q(S_t,A_t)\right)$$

Forward View Sarsa($\lambda$)

As before, we saw that the n-step return itself is noisy and sensitive to hyperparameter choice of n and $\alpha$. So the better way is to average the value over all $n$ steps.

• The $q^{\lambda }$ return combines all $n$-step Q-returns $q_t^{(n)}$
• Using weight $(1-\lambda ){\lambda }^{n-1}$, we have: $$q_t^{\lambda }=(1-\lambda )\sum _{n=1}^{\infty }{\lambda }^{n-1}{q}_t^{(n)}$$
• And the forward view Sarsa($\lambda$) is: $$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha \left(q_t^{\lambda }-Q(S_t,A_t)\right)$$

Backward View Sarsa($\lambda$)

Recall that we used eligibility traces to construct the backward view TD($\lambda$). As the forward view algo is not an online policy - we need to wait until the end of the episode to do the update.

• Just like TD($\lambda$), we use eligibility traces in an online algorithm
• But Sarsa($\lambda$) has one eligibility trace for each state-action pair instead of just for every state $$\begin{array}{rl}{E}_0(s,a)& =0\\ E_t(s,a)& =\gamma \lambda E_{t-1}(s,a)+1(S_t=s,A_t=a)\end{array}$$
• $Q(s,a)$ is updated for every state $s$ and action $a$
• In proportion to TD-error ${\delta }_t$ and eligibility trace $E_t(s,a)$: $$\begin{array}{rl}{\delta }_t& =R_{t+1}+\gamma Q(S_{t+1},A_{t+1})-Q(S_t,A_t)\\ Q(s,a)& \leftarrow Q(s,a)+\alpha {\delta }_t{E}_t(s,a)\end{array}$$

Algorithm. Sarsa($\lambda$) On Policy Algorithm.

• Initialize $Q(s,a)$ arbitrarily, for all $s\in \mathcal{S},a\in \mathcal{A}$
• Repeat (for each episode):
  • $E(s,a)=0$, for all $s\in \mathcal{S},a\in \mathcal{A}$
  • Initialize $S,A$
  • Repeat (for each step of episode):
    • Take action $A$, observe $R,S^{\prime }$
    • Choose $A^{\prime }$ from $S^{\prime }$, using policy derived from $Q$ (E.g. $ϵ$-greedy)
    • $\delta \leftarrow R+\gamma Q(S^{\prime },A^{\prime })-Q(S,A)$
    • $E(S,A)\leftarrow E(S,A)+1$
    • For all $s\in \mathcal{S},a\in \mathcal{A}$:
      • $Q(s,a)\leftarrow Q(s,a)+\alpha \delta E(s,a)$
      • $E(s,a)\leftarrow \gamma \lambda E(s,a)$
    • $S\leftarrow S^{\prime },A\leftarrow A^{\prime }$
  • Until $S$ is terminal

Note that for a given step we have a single value of $\delta$ which is our TD error, but we propagate that to all $s,a$ pairs based on the eligibility trace, as potentially every $s,a$ pair could have contributed to it.

Off Policy Learning

So far we have been looking at on-policy learning. However it is often useful to do off policy learning, i.e. evaluate a target policy $\pi (a|s)$ to compute $v_{\pi }(s)$ or $q_{\pi }(s,a)$, while we follow the behaviour policy $\mu (a|s):\{S_1,A_1,R_2,...,S_T\sim \mu \}$. Of course in this case, $\mu \ne \pi$.

Why is off policy learning useful?

• We can learn from observing humans or other agents
• We can re-use experience that was previously generated from old policies ${\pi }_1,{\pi }_2,...{\pi }_{t-1}$, possibly in a batched manner
• We can learn about the optimal policy while following the exploratory policy
• We can learn about multiple policies while following one policy

First mechanism is importance sampling. The main idea is to estimate the expectation of a different distribution by re-weighting the distributions: $$\begin{array}{rl}{\mathbb{E}}_{X\sim P}[f(X)]& =\sum P(X)f(X)\\ & =\sum Q(X)\frac{P(X)}{Q(X)}f(X)\\ & ={\mathbb{E}}_{X\sim Q}\left[\frac{P(X)}{Q(X)}f(X)\right]\end{array}$$

We can apply importance sampling to Monte Carlo for Off policy monte carlo learning:

• We use returns generated from behaviour policy $\mu$ to evaluate $\pi$
• Then we weight the return $G_t$ according to the ratio of probabilities between the two policies
• We need to apply the correction at every time step along the whole episode, because the change in policy affects every time step $$G_t^{\pi /\mu }=\frac{\pi (A_t|S_t)}{\mu (A_t|S_t)}\frac{\pi (A_{t+1}|S_{t+1})}{\mu (A_{t+1}|S_{t+1})}...\frac{\pi (A_T|S_T)}{\mu (A_T|S_T)}{G}_t$$
• And then update the value towards the corrected return $$V(S_t)\leftarrow V(S_t)+\alpha \left(G_t^{\pi /\mu }-V(S_t)\right)$$

While off policy MC learning is theoretically sound, there are some major problems which make it practically useless in practice:

• Importance sampling dramatically increases variance, as we are adjusting over every time step, and the cumulative effect over the whole episode makes our estimate of $G_t^{\pi /\mu }$ vary wildly
• We also cannot use this adjustment if $\mu$ is zero when $\pi$ is non-zero

So we have to use bootstrapping for importance sampling. This allows us to only adjust the probability for one time step. So we have importance sampling for off policy TD:

• We use TD targets generated from $\mu$ to evaluate $\pi$
• For TD(0), We weight the TD target $R+\gamma V(S^{\prime })$ by importance sampling
• This means we only need a single importance sampling correction: $$V(S_t)\leftarrow V(S_t)+\alpha \left(\frac{\pi (A_t|S_t)}{\mu (A_t|S_t)}(R_{t+1}+\gamma V(S_{t+1}))-V(S_t)\right)$$
• This has much lower variance that MC importance sampling, and could work if $\mu$ and $\pi$ do not differ by too much over a single step

As we have seen, importance sampling leads to large variances. The best solution is known as Q-learning, which is specific to TD(0) or Sarsa(0).

• Does not require any importance sampling
• Allows off policy learning of action values $Q(s,a)$

Recall that $\mu$ is the behaviour policy that our agent is actually following, and $\pi$ is a target policy that we want to learn from. The main idea is that in our Sarsa(0) update step, we update the Q-value towards the target policy $\pi$, but allow our agent to continue following the behaviour policy $\mu$.

This allows the agent to explore the environment using $\mu$, but learn from the action-value function of $\pi$. Specifically:

• We choose each next action for the agent using behaviour policy $A_{t+1}\sim \mu (.|S_t)$
• But use alternative successor action $A^{\prime }\sim \pi (.|S_t)$ in our Q-value update
• So we update $Q(S_t,A_t)$ using $A^{\prime }$: $$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha (R_{t+1}+\gamma Q(S_{t+1},A^{\prime })-Q(S_t,A_t))$$

Note importantly that we are using $A^{\prime }\sim \pi$ in the Q value above, instead of $A_{t+1}$. This allows us to learn off policy.

Q-Learning (or SARSA-MAX)

A special case of Q-learning is the case where the target policy is greedy wrt $Q(s,a)$. This is usually what people refer to as Q-learning.

We allow both behaviour and target policies to improve:

• The target policy $\pi$ is greedy wrt $Q(s,a)$, i.e. $$\pi (S_{t+1})=\underset{a^{\prime }}{\mathrm{argmax}}Q(S_{t+1},a^{\prime })$$
• The behaviour policy $\mu$ is $ϵ$-greedy wrt $Q(s,a)$ again
• The learning target inside the Q-update then simplifies as follows: $$\begin{array}{rl}& R_{t+1}+\gamma Q(S_{t+1},A^{\prime })\\ & =R_{t+1}+\gamma Q(S_{t+1},\underset{a^{\prime }}{\mathrm{argmax}}Q(S_{t+1},a^{\prime }))\\ & =R_{t+1}+\underset{a^{\prime }}{\max }\gamma Q(S_{t+1},a^{\prime })\end{array}$$

Note that since we are following a greedy target policy, the action chosen will be the Q-maximizing one (line 2). Since we are choosing the Q-maximizing action, we get the maximum Q-value over all possible actions (line 3). This simplifies the equation quite abit, and now it resembles the Bellman optimality equation.

This leads us to the well known Q-learning algorithm, which David calls Sarsa-max. The Q-update is: $$Q(S,A)\leftarrow Q(S,A)+\alpha \left(R+\gamma \underset{a^{\prime }}{\max }Q(S^{\prime },a^{\prime })-Q(S,A)\right)$$

There is a theorem that tells us that the Q-learning control algorithm converges to the optimal action-value function, i.e. $Q(s,a)\to q_{\ast }(s,a)$

To wrap up, here is a classification of some algorithms we have so far: