Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dispelling misconceptions about RLHF

aerial-toothpaste-34a.notion.site

Post reply on HN
Print view View original

11–20 of 35 posts

Re: Dispelling misconceptions about RLHF

#11

Post by andy99 »

vertere wrote: post #10

I'm confused about their definition of RL. > ... SFT is a subset of RL. > The first thing to note about traditional SFT is that the responses in the examples are typically human written. ... But it is also possible to build the dataset using responses from the model we’re about to train. ... This is called Rejection Sampling. I can see why someone might say there's overlap between RL and SFT (or semi-supervised FT),…

A couple things I've seen go by that make the connection. I haven't looked at them closely enough to have an opinion.

> https://arxiv.org/abs/2507.12856

> https://justinchiu.netlify.app/blog/sftrl/

Re: Dispelling misconceptions about RLHF

#12

Post by einrealist »

> “Successful” is importantly distinct from “correct.”

This is the most important sentence describing the fundamental issue that LLMs have. This severely limits the technology's useful applications. Yet OpenAI and others constantly lie about it.

The article very clearly explains why models won't be able to generalise unless RL is performed constantly. But that's not scalable, has other problems in itself. For example, it still runs into paradoxes where the training mechanism has to know the answer in order to formulate the question. (This is precisely where the concept of World Models comes in or why symbolism becomes more important.)

LLMs perform well in highly specialised scenarios with a well-defined and well-known problem space. It's probably possible to increase accuracy and correctness by using lots of interconnected models that can perform RL with each other. Again, this raises questions of scale and feasibility. But I think our brains (together with the other organs) work this way.

Re: Dispelling misconceptions about RLHF

#14

Post by ProofHouse »

williamtrask wrote: post #13

Nit: the author says that supervised fine tuning is a type of RL, but it is not. RL is about delayed reward. Supervised fine tuning is not in any way about delayed reward.

Well they can be used together in some contexts so while they are different, you could also say RL can help Supervised Fine Tuning for further optimization

Re: Dispelling misconceptions about RLHF

#15

Post by abatilo »

josh-sematic wrote: post #9

The mechanisms the author describe are used for RLHF, but are not sufficient for training the recent slew of “reasoning models.” To do that, you have to generate rewards not based on proximity to some reference full answer transcript, but rather based on how well the final answer (ex: the part after the “thinking tokens”) meets your reward criteria. This turns out to be a lot harder to do than the mechanisms used for…

FWIW, that was Karpathy, not Sutskever:

https://x.com/karpathy/status/1835561952258723930?s=19

Re: Dispelling misconceptions about RLHF

#17

Post by [deleted] »

josh-sematic wrote: post #9

The mechanisms the author describe are used for RLHF, but are not sufficient for training the recent slew of “reasoning models.” To do that, you have to generate rewards not based on proximity to some reference full answer transcript, but rather based on how well the final answer (ex: the part after the “thinking tokens”) meets your reward criteria. This turns out to be a lot harder to do than the mechanisms used for…

[deleted]

Re: Dispelling misconceptions about RLHF

#18

Post by Foreignborn »

einrealist wrote: post #12

> “Successful” is importantly distinct from “correct.” This is the most important sentence describing the fundamental issue that LLMs have. This severely limits the technology's useful applications. Yet OpenAI and others constantly lie about it. The article very clearly explains why models won't be able to generalise unless RL is performed constantly. But that's not scalable, has other problems in itself. For example…

can you say more about world models or symbolism?

i thought world models like genie 3 would be the training mechanism, but i likely misunderstand.

Re: Dispelling misconceptions about RLHF

#19

Post by mehulashah »

This article is really two. One that describes RL. The other is how they applied it. The former was quite helpful because it demystified much of the jargon that I find in AI. All branches of science have jargon. I find the AI ones especially impenetrable.

Re: Dispelling misconceptions about RLHF

#20

Post by schlipity »

The site is designed poorly and is stopping me from reading the article. I use NoScript, and it immediately redirects me to a "Hey you don't have javascript enabled, please enable it to read" page that is on a different domain from the website the article is on. I tried to visit notion.site to try and whitelist it temporarily, but it redirects back to notion.so and notion.com.

Rather than jump through more hoops, I'm just going to give up on reading this one.

Post reply on HN