331–335 of 335 posts
Earlier quoted context omitted.
Thank you for providing an arxiv! An aside, I finally do appreciate single column format now, makes it easier to convert to epub.
When you are done with the section on RLVR, consider whether the model is predicting tokens, or making moves. There is a reason the word "policy" is used in RL.
Earlier quoted context omitted.
Hey man, I'm not crazy. Yes, the goal is to maximize the cumulative reward like you say but to do that a policy (agent) has to take the actions that maximize its expected return in each time step. That's what the discount factor applies to, the expected return. To maximize its expected return it has to predict that taking action α in state s in time step t will produce state s' and reward r in time step t+1. Which bt…
Your initial comment says "reward". Reward and return are not the same thing. The policy is choosing the moves based off of returns at the next state, not the immediate rewards. One choice might have reward 0 and expected return 100. Another reward 10 and expected return 20. And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate e…
Earlier quoted context omitted.
Hey man, I'm not crazy. Yes, the goal is to maximize the cumulative reward like you say but to do that a policy (agent) has to take the actions that maximize its expected return in each time step. That's what the discount factor applies to, the expected return. To maximize its expected return it has to predict that taking action α in state s in time step t will produce state s' and reward r in time step t+1. Which bt…
Your initial comment says "reward". Reward and return are not the same thing. The policy is choosing the moves based off of returns at the next state, not the immediate rewards. One choice might have reward 0 and expected return 100. Another reward 10 and expected return 20. And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate e…
>> And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.
I think rather that's the job of the value function, to inform the policy of the, well, value of a state, or state-action pair. Somehow the policy needs to know what a good action is to take in the current state, or what is the best action to take. And of course it doesn't have to run the entire episode before it does that: but that's what I mean that it predicts the next move and not just the outcome of the entire game.
Earlier quoted context omitted.
Your initial comment says "reward". Reward and return are not the same thing. The policy is choosing the moves based off of returns at the next state, not the immediate rewards. One choice might have reward 0 and expected return 100. Another reward 10 and expected return 20. And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate e…
You're right of course and my bad but in my defense even Sutton & Barto lapse into using the term interchangeably (I caught them at it while reading the book earlier). >> And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that re…
Earlier quoted context omitted.
Sure, you're right. But it's a policy learned from a next-token prediction task. You could also call it an inferrer or generator or whatever. The point is that it takes as input a sequence of preceding tokens and emits one more token to continue the sequence.
The policy is not learned token by token during RLHF and RLVR. The reward model doesn't score token by token.