Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Little Book of Reinforcement Learning

github.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: The Little Book of Reinforcement Learning

#21

Post by fxwin »

barrenko wrote: post #15
relyks wrote: post #7

Earlier quoted context omitted.

I'm assuming it's more in line with The Little Schemer series of books ( https://felleisen.org/matthias/BTLS-index.html ) or maybe the little book of deep learning ( https://fleuret.org/francois/lbdl.html )?

Proobably the later.

definitely the latter, it is even referenced in the foreword:

> Its goal is not to be exhaustive, but rather minimalist and easy to read. For this reason, it follows the format of The Little Book of Deep Learning [Fleuret 2023]. Its tone, however, is closer to that of a blog post, as the book is built around a single narrative thread. Its structure broadly follows that of Sutton and Barto’s Reinforcement Learning: An Introduction [Sutton et al. 2018], which remains the canonical reference on the subject.

Re: The Little Book of Reinforcement Learning

#24

Post by porridgeraisin »

ainch wrote: post #19
programjames wrote: post #11

I skimmed through the book, and it's lacking the information theory foundations. For example, "trust region methods" come from maximizing the policy's relative entropy (to a reference policy) under a tournament system where high-scoring agents are exponentially likely to survive. In general, a reward is the negative bits it costs an environment to propagate an agent (multiplied by some temperature).

Do you have a good source on this information theory framing? I don't remember it being covered in Sutton & Barto.

It's just another way to frame it. It's as foundational as the many other ways to frame it. I'm not aware of any major insight you get specifically from this framing. Is there one?

Re: The Little Book of Reinforcement Learning

#25

Post by porridgeraisin »

janalsncm wrote: post #18

I wonder what Sutton thinks about some of the more recent innovations in RL like GRPO. In some ways it’s new, in other ways it’s an echo of RLOO.

GRPO is policy gradient/PPO with your value function baseline monte carlo estimated using k rollouts. The only new thing is finding out it works well with binary rewards and LLM policies.

Re: The Little Book of Reinforcement Learning

#27

Post by janalsncm »

porridgeraisin wrote: post #25
janalsncm wrote: post #18

I wonder what Sutton thinks about some of the more recent innovations in RL like GRPO. In some ways it’s new, in other ways it’s an echo of RLOO.

GRPO is policy gradient/PPO with your value function baseline monte carlo estimated using k rollouts. The only new thing is finding out it works well with binary rewards and LLM policies.

It is a huge improvement to PPO because you don’t need a separate critic model which cuts memory costs in half and stabilizes training.

Re: The Little Book of Reinforcement Learning

#28

Post by porridgeraisin »

janalsncm wrote: post #27
porridgeraisin wrote: post #25

Earlier quoted context omitted.

GRPO is policy gradient/PPO with your value function baseline monte carlo estimated using k rollouts. The only new thing is finding out it works well with binary rewards and LLM policies.

It is a huge improvement to PPO because you don’t need a separate critic model which cuts memory costs in half and stabilizes training.

Yes, but monte carlo estimating the critic model is not new.

Re: The Little Book of Reinforcement Learning

#29

Post by newsomix9xl »

herodoturtle wrote: post #16
newsomix9xl wrote: post #10

Real biological operant behavior isn't exactly trial and error learning. Many factors shape and guide initial responses. What I've noticed in some descriptions of models is the use of optimization for reinforcement to shape responses. In real organisms behavior may be controlled by short or long term outcomes, and may oscillate between this "optimization" based on schedules. This produces variability in the trials wh…

I found this comment/question deeply intriguing. I’m no expert at this and was wondering what you meant by the following: > In real organisms behavior may be controlled by short or long term outcomes, and may oscillate between this "optimization" based on schedules Could you perhaps provide an example that would help me understand what you mean? Thanks for the insightful comment either way.

In humans this is often called impulsivity, the preference for smaller sooner outcomes. This is often seen in children and animals and in some adult human behavior.

An impulsive choice is not optimal. You can buy a cheaper pack of gum at Costco in a week or get one for three times the cost right now.

Post reply on HN