Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mathematical Foundations of Reinforcement Learning

github.com

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Mathematical Foundations of Reinforcement Learning

#31

Post by al_th »

eachro wrote: post #26

During the openai gym era of RL, one of the great selling pts was that RL was very approachable for a new comer as the gym environments were small and tractable that a hobbyist could learn a little bit of RL, try it out on cartpole and see how it'd perform. Are there similarly tractable RL tasks/learning environments with LLMs? From the outside, my impression is that you need some insane GPU access to even start to m…

This is entirely doable.

I'm absolutely not versed in RL, but I wanted to understand GRPO, the RL algorithm behind Deepseek's latest model.

I started from a very simple LLM, inspired from Andrej Karpathy's "GPT from scratch" video (https://www.youtube.com/watch?v=kCc8FmEb1nY). Then, I added onto that the GRPO algorithm, which in itself is very simple.

I made a GitHub repo if you want to try it out : https://github.com/Al-th/grpo_experiment

Re: Mathematical Foundations of Reinforcement Learning

#32

Post by sigmoid10 »

Culonavirus wrote: post #30

> This book, however, requires the reader to have some knowledge of probability theory and linear algebra. This is so funny to me, I see it often and I'm always like "yea, right, some knowledge"... these statements always need to be taken with a grain of salt and an understanding that math nerds wrote them. Average programmers with average math skills (like me) beware ;)

This usually means that average CS or EE university level students should be able to easily follow it even if they have never touched the topic. It's far below the level of math and physics degrees, but still somewhat above what you could expect from an average self taught programmer.

Re: Mathematical Foundations of Reinforcement Learning

#33

Post by 363849473754 »

al_th wrote: post #31
eachro wrote: post #26

During the openai gym era of RL, one of the great selling pts was that RL was very approachable for a new comer as the gym environments were small and tractable that a hobbyist could learn a little bit of RL, try it out on cartpole and see how it'd perform. Are there similarly tractable RL tasks/learning environments with LLMs? From the outside, my impression is that you need some insane GPU access to even start to m…

This is entirely doable. I'm absolutely not versed in RL, but I wanted to understand GRPO, the RL algorithm behind Deepseek's latest model. I started from a very simple LLM, inspired from Andrej Karpathy's "GPT from scratch" video ( https://www.youtube.com/watch?v=kCc8FmEb1nY ). Then, I added onto that the GRPO algorithm, which in itself is very simple. I made a GitHub repo if you want to try it out : https://github.…

GRPO project is neat. Would you be willing to do a Karpathy-style explainer, breaking down the algorithm from scratch? It’s hard to understand on its own without prior background knowledge.

Re: Mathematical Foundations of Reinforcement Learning

#34

Post by auggierose »

almostgotcaught wrote: post #29
jgord wrote: post #28

Earlier quoted context omitted.

Sure.. and neural networks came out a very long time ago, but are now arguably approaching usefulness in LLMs. Perhaps thats because it takes a while for the ideas to get polished/weeded and diffuse into the engineer zeitgeist .. or it could be that compute / GPUs are now powerful enough to run at the scale needed. re : "RL cannot be used to solve real world problems" .. well, I would argue that these are useful real…

> Come back and chastize me if we dont see more RL startups in 12 months time ! 1999 is 26 years ago but ya sure this is the year they finally take off. > Perhaps thats because it takes a while for the ideas to get polished/weeded and diffuse into the engineer zeitgeist .. or it could be that compute / GPUs are now powerful enough to run at the scale needed. Or perhaps it could be that you're wrong and they're useles…

I think you GOT caught here. That's why you don't respond to the Nobel prize winning example of RL.

Re: Mathematical Foundations of Reinforcement Learning

#35

Post by bglazer »

auggierose wrote: post #34
almostgotcaught wrote: post #29

Earlier quoted context omitted.

> Come back and chastize me if we dont see more RL startups in 12 months time ! 1999 is 26 years ago but ya sure this is the year they finally take off. > Perhaps thats because it takes a while for the ideas to get polished/weeded and diffuse into the engineer zeitgeist .. or it could be that compute / GPUs are now powerful enough to run at the scale needed. Or perhaps it could be that you're wrong and they're useles…

I think you GOT caught here. That's why you don't respond to the Nobel prize winning example of RL.

Are we talking about AlphaFold? It did not use RL, right?

Re: Mathematical Foundations of Reinforcement Learning

#36

Post by auggierose »

bglazer wrote: post #35
auggierose wrote: post #34

Earlier quoted context omitted.

I think you GOT caught here. That's why you don't respond to the Nobel prize winning example of RL.

Are we talking about AlphaFold? It did not use RL, right?

I think it does: https://juanraul8.github.io/master-praktikum/

Re: Mathematical Foundations of Reinforcement Learning

#37

Post by RiDiracTid »

jgord wrote: post #18

Highly recommended .. even the main contents diagram is a great visual overview of RL in general, as is the 30 minute intro YT video. Im expecting to see a lot of hyper growth startups using RL to solve a realworld problem in engineering / logistics / medicine LLMs currently attract all the hype for good reasons, but Im surprised VCs dont seem to be looking at RL companies specifically.

RL is definitely really cool but I heavily doubt that we're gonna see 'hyper growth' from RL outside of the context of maybe training reasoning LLMs.

The period from ~2012-2019 of AI research had deepmind (who was the undisputed leader in money and talent) go all in on RL to solve problems and while they did do lots of interesting and useful work, there wasn't anything quite so extraordinary / revolutionary in massively accelerating the field or some sort of crazy breakthrough.

Their over-focus on RL instead of transformers/llms is what allowed OpenAI to surprise everyone and overtake deepmind.

Yes, RL is a useful tool, but outside the context of training LLMs for reasoning there isn't really any breakthrough that makes it more than an interesting tool for certain situations.

Re: Mathematical Foundations of Reinforcement Learning

#38

Post by bitvoid »

almostgotcaught wrote: post #25
jgord wrote: post #18

Highly recommended .. even the main contents diagram is a great visual overview of RL in general, as is the 30 minute intro YT video. Im expecting to see a lot of hyper growth startups using RL to solve a realworld problem in engineering / logistics / medicine LLMs currently attract all the hype for good reasons, but Im surprised VCs dont seem to be looking at RL companies specifically.

> Im expecting to see a lot of hyper growth startups using RL to solve a realworld problem in engineering / logistics / medicine I love when people on hn make market predictions based on how revolutionary they think something is. I guess startup people thank they're also VC people. FYI Sutton's book came out in 1999; none of this is revolutionary anymore and yet I don't see any "hyper growth". The reason is exactly b…

> you cannot use them to solve real world problems

Doesn't waymo and other self-driving systems use reinforcement learning? I thought it was used in robotics as well (i.e., bipedal, quadrupedal movement).

Re: Mathematical Foundations of Reinforcement Learning

#40

Post by currymj »

363849473754 wrote: post #33
al_th wrote: post #31

Earlier quoted context omitted.

This is entirely doable. I'm absolutely not versed in RL, but I wanted to understand GRPO, the RL algorithm behind Deepseek's latest model. I started from a very simple LLM, inspired from Andrej Karpathy's "GPT from scratch" video ( https://www.youtube.com/watch?v=kCc8FmEb1nY ). Then, I added onto that the GRPO algorithm, which in itself is very simple. I made a GitHub repo if you want to try it out : https://github.…

GRPO project is neat. Would you be willing to do a Karpathy-style explainer, breaking down the algorithm from scratch? It’s hard to understand on its own without prior background knowledge.

Find materials on PPO which should be widespread since it is the most popular RL algorithm. GRPO works on the same principles, just makes certain estimates from samples rather than training an auxiliary neural network to make them.
Post reply on HN