Sandboxxxx

According to the question the weights 𝐰 are initialized as:

• $\mathbf{w}=⟨1,1,1,1,1,1,5⟩$

According to the question, we update weights 𝐰 using semi-gradient TD(0):

• $\mathbf{w}\leftarrow \mathbf{w}+\alpha [R+\gamma \hat{v}(S^{\prime },\mathbf{w})-\hat{v}(S,\mathbf{w})]\nabla \hat{v}(S,\mathbf{w})$

According to the question, 𝛼=0.1 𝛾=0.95 𝑅=0, thus:

• $\mathbf{w}\leftarrow \mathbf{w}+0.1\ast [0.95\ast \hat{v}(S^{\prime },\mathbf{w})-\hat{v}(S,\mathbf{w})]\nabla \hat{v}(S,\mathbf{w})$

According to the question, the diagram contains 6 states/nodes, thus 𝑣̂(𝑆,𝐰) is defined as:

• $\hat{v}(s_1,\mathbf{w})=w_0+2w_1$
• $\hat{v}(s_2,\mathbf{w})=w_0+2w_2$
• $\hat{v}(s_3,\mathbf{w})=w_0+2w_3$
• $\hat{v}(s_4,\mathbf{w})=w_0+2w_4$
• $\hat{v}(s_5,\mathbf{w})=w_0+2w_5$
• $\hat{v}(s_6,\mathbf{w})=2w_0+w_6$

According to the question, each transition/arrow is denoted as 1 training example in the batch. Thus they are a total of 6 training examples.

1. 𝑠₁ → 𝑠₆
2. 𝑠₂ → 𝑠₆
3. 𝑠₃ → 𝑠₆
4. 𝑠₄ → 𝑠₆
5. 𝑠₅ → 𝑠₆
6. 𝑠₆ → 𝑠₆

So given training example #1 (𝑠₁ → 𝑠₆) the update becomes:

• $\mathbf{w}\leftarrow \mathbf{w}+0.1\ast [0.95\ast (2w_0+w_6)-(w_0+2w_1)]\nabla (w_0+2w_1)$

Since this is a batch update, 𝐰 will NOT be updated until all training examples have been processed:

• $0.1\ast [0.95\ast (2\ast 1+5)-(1+2\ast 1)]\nabla (w_0+2w_1)$
• $0.365\ast \nabla (w_0+2w_1)$
• $0.365\ast [1,2,0,0,0,0,0]$
• $[0.365,0.730,0,0,0,0,0]$

Do this for the rest of the training samples, add them up and update the weight vector 𝐰 accordingly.