Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RLHF Book

rlhfbook.com

Post reply on HN
Print view View original

21–30 of 41 posts

Re: RLHF Book

#21

Post by noch »

brcmthrowaway wrote: post #9
tintor wrote: post #8

Earlier quoted context omitted.

They are different processes. - RLHF: Turns pre-trained model (which just performs autocomplete of text) into a model that you can speak with, ie. answer user questions and refuse providing harmful answers. - Distillation: Transfer skills / knowledge / behavior from one model (and architecture) to a smaller model (and possibly different architecture), by training second model on output log probs of first model.

So RLHF is the secret sauce behind modern LLMs?

> So RLHF is the secret sauce behind modern LLMs?

Karpathy wrote[^0]:

"

RL is powerful. RLHF is not.

[…]

And yet, RLHF is a net helpful step of building an LLM Assistant. I think there's a few subtle reasons but my favorite one to point to is that through it, the LLM Assistant benefits from the generator-discriminator gap. That is, for many problem types, it is a significantly easier task for a human labeler to select the best of few candidate answers, instead of writing the ideal answer from scratch.

[…]

No production-grade actual RL on an LLM has so far been convincingly achieved and demonstrated in an open domain, at scale.

"

---

[^0]: https://x.com/karpathy/status/1821277264996352246

Re: RLHF Book

#22

Post by littlestymaar »

bryan0 wrote: post #12
alexhutcheson wrote: post #5

Earlier quoted context omitted.

DeepSeek-R1 had an RLHF step in their post-training pipeline (section 2.3.4 of their technical report[1]). In addition, the "reasoning-oriented reinforcement learning" step (section 2.3.2) used an approach that is almost identical to RLHF in theory and implementation. The main difference is that they used a rule-based reward system, rather than a model trained on human preference data. If you want to train a model li…

Yes but these were steps were not used in R1-zero where its reasoning capabilities were trained.

And as a result R1-zero is way too crude to be used directly, which is a good indication that it remains relevant.

Re: RLHF Book

#23

Post by drmindle12358 »

kadushka wrote: post #4
alexhutcheson wrote: post #3

Glad to see the author making a serious effort to fill the gap in public documentation of RLHF theory and practice. The current state of the art seems to be primarily documented in arXiv papers, but each paper is more like a "diff" than a "snapshot" - you need to patch together the knowledge from many previous papers to understand the current state. It's extremely valuable to "snapshot" the current state of the art i…

Has r1 made RLHF obsolete?

You meant to ask "Has r1 made SFT obsolete?" ?

Re: RLHF Book

#24

Post by JTyQZSnP3cQGa8B »

tintor wrote: post #8
brcmthrowaway wrote: post #7

Whats the difference between RLHF and distillation?

They are different processes. - RLHF: Turns pre-trained model (which just performs autocomplete of text) into a model that you can speak with, ie. answer user questions and refuse providing harmful answers. - Distillation: Transfer skills / knowledge / behavior from one model (and architecture) to a smaller model (and possibly different architecture), by training second model on output log probs of first model.

> answer user questions and refuse providing harmful answers.

I wonder why this thing can have so much hype. Here is the NewGCC, it's a binary only compiler that refuses to compile applications that it doesn't like... What happened to all the hackers that helped create the open-source movement? Where are they now?

Re: RLHF Book

#25

Post by parodysbird »

noch wrote: post #21
brcmthrowaway wrote: post #9

Earlier quoted context omitted.

So RLHF is the secret sauce behind modern LLMs?

> So RLHF is the secret sauce behind modern LLMs? Karpathy wrote[^0]: " RL is powerful. RLHF is not. […] And yet, RLHF is a net helpful step of building an LLM Assistant. I think there's a few subtle reasons but my favorite one to point to is that through it, the LLM Assistant benefits from the generator-discriminator gap. That is, for many problem types, it is a significantly easier task for a human labeler to selec…

RL on any production system is very tricky, and so it seems difficult to work in any open domain, not just LLMs. My suspicion is that RL training is a coalgebra to almost every other form of ML and statistical training, and we don't have a good mathematical understanding how it behaves.

Re: RLHF Book

#26

Post by cratermoon »

alexhutcheson wrote: post #3

Glad to see the author making a serious effort to fill the gap in public documentation of RLHF theory and practice. The current state of the art seems to be primarily documented in arXiv papers, but each paper is more like a "diff" than a "snapshot" - you need to patch together the knowledge from many previous papers to understand the current state. It's extremely valuable to "snapshot" the current state of the art i…

Is there not a survey paper on RLHF equivalent to the "A Survey on Large Language Model based Autonomous Agents" paper? Someone should get on that.

Re: RLHF Book

#27

Post by cratermoon »

gr3ml1n wrote: post #17
alexhutcheson wrote: post #3

Glad to see the author making a serious effort to fill the gap in public documentation of RLHF theory and practice. The current state of the art seems to be primarily documented in arXiv papers, but each paper is more like a "diff" than a "snapshot" - you need to patch together the knowledge from many previous papers to understand the current state. It's extremely valuable to "snapshot" the current state of the art i…

SFT can be used to give negative feedback/examples. That's one of the lesser-known benefits/tricks of system messages. E.g: System: You are a helpful chatbot. User: What is 1+1? Assistant: 2. And System: You are terrible at math. User: What is 1+1? Assistant: 0.

    System: It's a lovely morning in the village and you are a horrible goose.
    User: Throw the rake into the lake

Re: RLHF Book

#28

Post by _giorgio_ »

https://arxiv.org/abs/2412.05265

Reinforcement Learning: An Overview Kevin Murphy

    This manuscript gives a big-picture, up-to-date overview of the field of (deep) reinforcement learning and sequential decision making, covering value-based RL, policy-gradient methods, model-based methods, and various other topics (including a very brief discussion of RL+LLMs).
From: Kevin Murphy [view email] [v1] Fri, 6 Dec 2024 18:53:49 UTC (6,099 KB)

Re: RLHF Book

#29

Post by _giorgio_ »

cratermoon wrote: post #26
alexhutcheson wrote: post #3

Glad to see the author making a serious effort to fill the gap in public documentation of RLHF theory and practice. The current state of the art seems to be primarily documented in arXiv papers, but each paper is more like a "diff" than a "snapshot" - you need to patch together the knowledge from many previous papers to understand the current state. It's extremely valuable to "snapshot" the current state of the art i…

Is there not a survey paper on RLHF equivalent to the "A Survey on Large Language Model based Autonomous Agents" paper? Someone should get on that.

*

1 point by _giorgio_ 0 minutes ago | next | edit | delete [–]

https://arxiv.org/abs/2412.05265

Reinforcement Learning: An Overview Kevin Murphy

    This manuscript gives a big-picture, up-to-date overview of the field of (deep) reinforcement learning and sequential decision making, covering value-based RL, policy-gradient methods, model-based methods, and various other topics (including a very brief discussion of RL+LLMs).
From: Kevin Murphy [view email] [v1] Fri, 6 Dec 2024 18:53:49 UTC (6,099 KB)
Post reply on HN