Value Function Approximation

Lecture 6: Value Function Approximation

This lecture looks into approximating functions with neural networks to overcome the large state-action space problem.

RL often encounters large problems:

• Backgammon: $10^{20}$ states
• Go: $10^{170}$ states
• Helicopter: continuous state space

We want to do policy evaluation and control efficiently in large state spaces. So far, we have represented $V$ or $Q$ with a lookup table:

• Every state $s$ has an entry $V(s)$
• Every state-action pair $s,a$ has an entry $Q(s,a)$

This is a problem for large MDPs:

• Too many states or actions to store in memory
• It is too slow or data inefficient to learn the value of each state individually

Solution for large MDPs:

• Estimate the value function with function approximation using parameters $w$:
  • $\hat{v}(s,w)\approx v_{\pi }(s)$
  • $\hat{q}(s,a,w)\approx q_{\pi }(s,a)$
• Generalizes from seen states to unseen states
• Update parameters of our function using MC or TD learning

Types of value function approximation (different architectures):

• Represent a given state $s$ with some parameters $w$. Then neural network spits out $\hat{v}(s,w)$, which is our value function for being in state $s$
• Have a neural network $\hat{q}(s,a,w)$, which takes in a state-action pair and spits out the Q value
• Sometimes, it is more efficient to have a neural network $\hat{q}(s,w)$, such that we feed in a single state and we get Q-values for every possible action in a single forward pass, i.e. we get $\hat{q}(s,a_1,w),\hat{q}(s,a_2,w),...$

Which function approximator? We focus on differentiable function approximators that we can easily optimize, i.e. Linear combinations of features, neural networks. Furthermore, we want a training algorithm for a non-iid, non-stationary set of data, so it is not standard supervised learning.

Incremental Methods

Gradient Descent

Starting with gradient descent.

• Let $J(w)$ be a differentiable function of parameter vector $w$
• Define the gradient of $J(w)$ to be a vector ${\nabla }_{w}J(w)$ where ${\nabla }_{w}J(w)[0]$ is $\frac{\partial J(w)}{\partial w_1}$
• To find the local minimum of $J(w)$, we adjust the parameter $w$ in the -ve gradient direction: $$△w=-\frac{1}{2}\alpha {\nabla }_{w}J(w)$$

Goal: find parameter vector $w$ minimizing mean squared error between approximate value fn $\hat{v}(s,w)$ and true oracle value fn $v_{\pi }(s)$ (assuming we know the oracle) $$J(w)={\mathbb{E}}_{\pi }[(v_{\pi }(S)-\hat{v}(S,w{)}^2)]$$

Gradient descent finds a local minimum: $$\begin{array}{rl}△w& =-\frac{1}{2}\alpha {\nabla }_{w}J(w)\\ & =\alpha {\mathbb{E}}_{\pi }[(v_{\pi }(S)-\hat{v}(S,w)){\nabla }_w\hat{v}(S,w)]\end{array}$$

Stochastic gradient descent samples the gradient: $$△w=\alpha (v_{\pi }(S)-\hat{v}(S,w)){\nabla }_w\hat{v}(S,w)$$

The nice thing about SGD is that it still converges under non-stationary environment. The expected update is equal to full gradient update.

Feature Vectors

To represent a state, we use a feature vector. $$\mathbf{x}(S)=\left(\begin{array}{c}{\mathbf{x}}_1(S)\\ ⋮\\ {\mathbf{x}}_n(S)\end{array}\right)$$

For example, the features (numeric) could be:

• Distance of robot to landmarks
• Trends in the stock market
• Configuration of pawn on a chess board

Linear Value Function Approximation

Let us represent the value function using a linear combination of features (i.e. just a dot product between two vectors): $$\hat{v}(S,w)=\mathbf{x}(S{)}^{T}w=\sum _{j=1}^n{\mathbf{x}}_j(S)w_j$$

The nice thing is that linear approximator is quadratic in the parameters $w$, so it is a convex optimization problem, i.e. SGD will converge on the global optimum: $$J(w)={\mathbb{E}}_{\pi }\left[(v_{\pi }(S)-\mathbf{x}(S{)}^{T}w{)}^2\right]$$

The gradient update is really simple: $$\begin{array}{rl}{\nabla }_w\hat{v}(S,w)& =\mathbf{x}(S)\\ △w& =\alpha \left(v_{\pi }(S)-\hat{v}(S,w)\right)\mathbf{x}(S)\end{array}$$

Note that we are just subbing the simple expression for ${\nabla }_w\hat{v}(S,w)$ into the general $△w$ formula above. The update may be interpreted as step-size x prediction error x feature value. This means that features with high correlation with the prediction error will have large gradient updates intuitively.

We can think of table lookup as a special case of linear value function approximation. Suppose we use a table lookup feature (1-hot) as follows: $${\mathbf{x}}^{table}(S)=\left(\begin{array}{c}1(S=s_1)\\ ⋮\\ 1(S=s_n)\end{array}\right)$$

And suppose we have a parameter vector of size $n$, such that we have one parameter for each state. Then we have: $$\hat{v}(S,w)=\left(\begin{array}{c}1(S=s_1)\\ ⋮\\ 1(S=s_n)\end{array}\right)\cdot \left(\begin{array}{c}{w}_1\\ ⋮\\ w_n\end{array}\right)$$

And we can see that this reduces to a table lookup where the parameter $w_j$ represents the state value for each state $j$.

Estimating the Oracle

So far, we have assumed that the true oracle value function $v_{\pi }(s)$ is available, but in RL there is no true label, only rewards. So in practice, we need to substitute a target for $v_{\pi }(s)$:

• For MC, the target is the return $G_t$: $$△w=\alpha (G_t-\hat{v}(S_t,w)){\nabla }_w\hat{v}(S_t,w)$$
• For TD(0), the target is the TD target $R_{t+1}+\gamma \hat{v}(S_{t+1},w)$: $$△w=\alpha (R_{t+1}+\gamma \hat{v}(S_{t+1},w)-\hat{v}(S_t,w)){\nabla }_w\hat{v}(S_t,w)$$
• For TD($\lambda$), the target is the $\lambda$-return $G_t^{\lambda }$: $$△w=\alpha (G_t^{\lambda }-\hat{v}(S_t,w)){\nabla }_w\hat{v}(S_t,w)$$

Monte Carlo with Value Function Approximation

We can think of our algorithm as supervised learning.

• Treat the return $G_t$ as an unbiased noisy sample of the true value $v_{\pi }(S_t)$
• We therefore are applying supervised learning to "training data": $$<S_t,G_1>,<S_2,G_2>,...,<S_T,G_T>$$
• For example, using linear MC policy evaluation: $$\begin{array}{rl}△w& =\alpha (G_t-\hat{v}(S_t,w)){\nabla }_w\hat{v}(S_t,w)\\ & =\alpha (G_t-\hat{v}(S_t,w))x(S_t)\end{array}$$
• MC evaluation converges to a local optimum even when using non-linear value function approximation

TD with Value Function Approximation

The same applies to TD learning, but we have some biased estimate:

• The TD target $R_{t+1}+\gamma \hat{v}(S_{t+1},w)$ is a biased sample of the true value $v_{\pi }(S_t)$ - it's biased because our own value function is a biased estimate
• We can still apply supervised learning to the "training data": $$\begin{gathered} <S_1,R_2+\gamma \hat{v}(S_2,w)>, \\ <S_2,R_3+\gamma \hat{v}(S_3,w)>, \\ ..., \\ <S_{t-1},R_T> \end{gathered}$$
• For example using linear TD(0): $$\begin{array}{rl}△w& =\alpha (R+\gamma \hat{v}(S^{\prime },w)-\hat{v}(S,w)){\nabla }_w\hat{v}(S,w)\\ & =\alpha \delta x(S)\end{array}$$
• There is a theorem showing that for linear TD(0), despite the bias, we will always converge (close) to the global optimum

Note: There is a little inconsistency in the above formula, once we start introducing bootstrapped approximations of the return. Recall that when we used the oracle $v_{\pi }$ to represent the target and took the derivative, only ${\nabla }_w\hat{v}(S,w)$ enters the derivative as we treat the oracle value as a constant.

However, once we introduce $\hat{v}$ itself to substitute the oracle function, we should technically include that term in the derivative as well. As it turns out, this is not a good idea and will not lead to convergence. There is some theoretical analysis for this to justify it.

TD($\lambda$) with Value Function Approximation

And again, we can do the same with TD-$\lambda$, since the $\lambda$-return $G_t^{\lambda }$ is also a biased sample of the true value $v_{\pi }(s)$:

• The training data is now: $$<S_1,G_1^{\lambda }>,<S_2,G_2^{\lambda }>,...,<S_{T-1},G_{T-1}^{\lambda }>$$
• The forward view linear TD($\lambda$) is: $$\begin{array}{rl}△w& =\alpha (G_t^{\lambda }-\hat{v}(S_t,w)){\nabla }_w\hat{v}(S_t,w)\\ & =\alpha (G_t^{\lambda }-\hat{v}(S_t,w))x(S_t)\end{array}$$
• The backward view linear TD($\lambda$) is: $$\begin{array}{rl}{\delta }_t& =R_{t+1}+\gamma \hat{v}(S_{t+1},w)-\hat{v}(S_t,w)\\ E_t& =\gamma \lambda E_{t-1}+x(S_t)\\ △w& =\alpha {\delta }_t{E}_t\end{array}$$
• There is a theorem to show that the forward view and backward view linear TD($\lambda$) are equivalent.

For the backward view, notice that the eligibility trace is now updated using the gradient wrt the parameter vector, namely ${\nabla }_w\hat{v}(S_t,w)$, which is of the same dimensionality as $w$. More precisely, the eligibility trace is the decaying accumulation of past gradients. In the linear case, this is an accumulation of the feature vector $x(S_t)$.

It is a bit unintuitive to understand why we use the accumulated gradient as the eligibility trace, but I suppose it is proved in the equivalence theorem between the forward and backward view. Perhaps we can just think of it as "the features which we see the most often will have high eligibility trace".

Control with Value Function Approximation

• Start with some random parameter vector $w$
• Set policy based on some greedy function $\pi =ϵ\text{-greedy}(q_w)$
• Do policy evaluation $\hat{q}(.,.,w)\approx q_{\pi }$

First we need to do everything again wrt to action-value function instead of value function to perform this algorithm. The steps are:

• Approximate the action-value function $$\hat{q}(S,A,w)\approx q_{\pi }(S,A)$$
• Minimize the mean squared error between approximate action value function and true oracle action value $q_{\pi }(S,A)$: $$J(w)={\mathbb{E}}_{\pi }\left[(q_{\pi }(S,A)-\hat{q}(S,A,w{)}^2)\right]$$
• Use SGD to find a local minimum: $$\begin{array}{rl}-\frac{1}{2}{\nabla }_{w}J(w)& =(q_{\pi }(S,A)-\hat{q}(S,A,w)){\nabla }_w\hat{q}(S,A,w)\\ △w& =\alpha (q_{\pi }(S,A)-\hat{q}(S,A,w)){\nabla }_w\hat{q}(S,A,w)\end{array}$$
• Again, we represent the state and action by a feature vector: $$\mathbf{x}(S,A)=\left(\begin{array}{c}{\mathbf{x}}_1(S,A)\\ ⋮\\ {\mathbf{x}}_n(S,A)\end{array}\right)$$
• Represent action value function by a linear combination of features: $$\hat{q}(S,A,w)=x(S,A{)}^{T}w=\sum _{j=1}^n{x}_j(S,A)w_j$$
• Do an SGD update: $$\begin{array}{rl}{\nabla }_w\hat{q}(S,A,w)& =x(S,A)\\ △w& =\alpha (q_{\pi }(S,A)-\hat{q}(S,A,w))x(S,A)\end{array}$$

Incremental Control Algorithms

Like prediction, we need to substitute a target for the unknown oracle $q_{\pi }(S,A)$. We sub out all the $v_{\pi }$ for an approximate target:

• For MC, target is the return $G_t$ $$△w=\alpha (G_t-\hat{q}(S_t,A_t,w)){\nabla }_w\hat{q}(S_t,A_t,w)$$
• For TD(0), the target is the TD target $R_{t+1}+\gamma Q(S_{t+1},A_{t+1}):$: $$△w=\alpha (R_{t+1}+\gamma \hat{q}(S_{t+1},A_{t+1},w))-\hat{q}(S_t,A_t,w){\nabla }_w\hat{q}(S_t,A_t,w)$$
• For forward view TD($\lambda$), the target is the action value $\lambda$-return: $$△w=\alpha (q_t^{\lambda }-\hat{q}(S_t,A_t,w)){\nabla }_w\hat{q}(S_t,A_t,w)$$
• For backward view TD($\lambda$), the equivalent update is: $$\begin{array}{rl}{\delta }_t& =R_{t+1}+\gamma \hat{q}(S_{t+1},A_{t+1},w)-\hat{q}(S_t,A_t,w)\\ E_t& =\gamma \lambda E_{t-1}+{\nabla }_w\hat{q}(S_t,A_t,w)\\ △w& =\alpha {\delta }_t{E}_t\end{array}$$

Should we bootstrap? Empirically across many examples, we almost always have the case that:

• MC takes too many steps because variance is too high
• TD(0) always has a large efficiency gain compared to MC
• There's always some $\lambda$ value in between which is better than TD(0)