I skimmed through the book, and it's lacking the information theory foundations. For example, "trust region methods" come from maximizing the policy's relative entropy (to a reference policy) under a tournament system where high-scoring agents are exponentially likely to survive. In general, a reward is the negative bits it costs an environment to propagate an agent (multiplied by some temperature).
The Little Book of Reinforcement Learning
11–20 of 29 posts
Re: The Little Book of Reinforcement Learning
#12Is this riffing on Strunk and Whites: The Elements of Style? Often referred to as "The Little Book".
I thought the title was an homage to François Fleuret's Little Book of Deep Learning.
Re: The Little Book of Reinforcement Learning
#13[dead]
Re: The Little Book of Reinforcement Learning
#14Is this riffing on Strunk and Whites: The Elements of Style? Often referred to as "The Little Book".
Most likely not: “The Little Book of …” has been a publisher’s standby since the nineteenth century (at least).
There are several "Libellus de Miraculis" (Little Book of Miracles) of different saints from the 12th century!
Re: The Little Book of Reinforcement Learning
#15Is this riffing on Strunk and Whites: The Elements of Style? Often referred to as "The Little Book".
I'm assuming it's more in line with The Little Schemer series of books ( https://felleisen.org/matthias/BTLS-index.html ) or maybe the little book of deep learning ( https://fleuret.org/francois/lbdl.html )?
Proobably the later.
Re: The Little Book of Reinforcement Learning
#16Real biological operant behavior isn't exactly trial and error learning. Many factors shape and guide initial responses. What I've noticed in some descriptions of models is the use of optimization for reinforcement to shape responses. In real organisms behavior may be controlled by short or long term outcomes, and may oscillate between this "optimization" based on schedules. This produces variability in the trials wh…
I found this comment/question deeply intriguing.
I’m no expert at this and was wondering what you meant by the following:
> In real organisms behavior may be controlled by short or long term outcomes, and may oscillate between this "optimization" based on schedules
Could you perhaps provide an example that would help me understand what you mean?
Thanks for the insightful comment either way.
Re: The Little Book of Reinforcement Learning
#17Is this riffing on Strunk and Whites: The Elements of Style? Often referred to as "The Little Book".
The Little Schemer, The Little Typer, The Little Reasoner, The Little Proover
The Little MLer ...
It has been going on for a while in Lispy land
Re: The Little Book of Reinforcement Learning
#18I wonder what Sutton thinks about some of the more recent innovations in RL like GRPO. In some ways it’s new, in other ways it’s an echo of RLOO.
Re: The Little Book of Reinforcement Learning
#19I skimmed through the book, and it's lacking the information theory foundations. For example, "trust region methods" come from maximizing the policy's relative entropy (to a reference policy) under a tournament system where high-scoring agents are exponentially likely to survive. In general, a reward is the negative bits it costs an environment to propagate an agent (multiplied by some temperature).
Do you have a good source on this information theory framing? I don't remember it being covered in Sutton & Barto.
Re: The Little Book of Reinforcement Learning
#20Real biological operant behavior isn't exactly trial and error learning. Many factors shape and guide initial responses. What I've noticed in some descriptions of models is the use of optimization for reinforcement to shape responses. In real organisms behavior may be controlled by short or long term outcomes, and may oscillate between this "optimization" based on schedules. This produces variability in the trials wh…
There is a field of hierarchical RL in which the optimisation occurs over a range of time scales/abstraction. But I'm not aware of much practical success for these approaches so far.