Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RLHF a LLM in <50 lines of Python

datadreamer.dev

Post reply on HN
Print view View original

21–30 of 68 posts

Re: RLHF a LLM in <50 lines of Python

#21

Post by nextaccountic »

espadrine wrote: post #18
patelajay285 wrote: post #12

Earlier quoted context omitted.

Yep, DPO is not technically “RL” and implicitly uses the LLM itself as a reward model, but training with DPO is far more stable for that reason.

DPO is as close to RL as RLHF. The latter also uses the LLM as a reward model. I'm not a fan of the RL/SL dichotomy, because the line gets so foggy. If you squint, every loss is a negative reward, and every policy improvement a supervised target. Still, what the code does isn't what is described in the paper that the page links to.

> I'm not a fan of the RL/SL dichotomy, because the line gets so foggy. If you squint, every loss is a negative reward, and every policy improvement a supervised target.

Isn't this just because reinforcement learning and supervised learning are both optimization problems?

Re: RLHF a LLM in <50 lines of Python

#22

Post by patelajay285 »

behnamoh wrote: post #20
jerpint wrote: post #2

I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”

This. If I'm the type of person who wants to do RLHF, then I'm the type of person who wants control and doesn't like delegating it to imported libraries.

This is built for ML researchers out of an academic lab. There's a ton of functionality in the library (beyond RLHF and alignment) that ML researchers do every day to write papers and run experiments that the library helps abstract and make repeatable and usable.

Unless your research hypothesis is specifically around improving or changing RLHF, it's unlikely you should be implementing it from scratch. Abstractions are useful for a reason. The library is quite configurable to let you tune any knobs you would want.

Re: RLHF a LLM in <50 lines of Python

#23

Post by espadrine »

nextaccountic wrote: post #21
espadrine wrote: post #18

Earlier quoted context omitted.

DPO is as close to RL as RLHF. The latter also uses the LLM as a reward model. I'm not a fan of the RL/SL dichotomy, because the line gets so foggy. If you squint, every loss is a negative reward, and every policy improvement a supervised target. Still, what the code does isn't what is described in the paper that the page links to.

> I'm not a fan of the RL/SL dichotomy, because the line gets so foggy. If you squint, every loss is a negative reward, and every policy improvement a supervised target. Isn't this just because reinforcement learning and supervised learning are both optimization problems?

In part, yes! But also because what used to define it was the human-curated datasets: SL contained input/output pairs, while RL contained episodes with sporadic rewards.

Nowadays, many datasets have different forms or are synthetic. DPO uses datasets with both positive and negative examples (instead of just a target output as with traditional SL); RLHF uses synthetic rewards.

Re: RLHF a LLM in <50 lines of Python

#24

Post by verticalscaler »

jerpint wrote: post #2

I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”

Yes you do. Most casuals are downright afraid of code. This messaging is meant to make the project more approachable.

Kind of like everybody knows the pop-science around e = mc^2 but most are completely oblivious that it takes a bunch of whiteboards to derive it and what all that actually means.

No pithy formula no way for the actual ideas to spread to the mainstream for you to somehow hear about it.

Re: RLHF a LLM in <50 lines of Python

#25

Post by vvrm »

jerpint wrote: post #2

I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”

Another problem with the title: the article is about DPO, which doesn’t do reinforcement learning. So not RLHF. I guess RLHF has more of a name recognition than DPO.

Re: RLHF a LLM in <50 lines of Python

#26

Post by patelajay285 »

vvrm wrote: post #25
jerpint wrote: post #2

I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”

Another problem with the title: the article is about DPO, which doesn’t do reinforcement learning. So not RLHF. I guess RLHF has more of a name recognition than DPO.

This was discussed in another comment, DPO is pretty much strictly better than RLHF + PPO, and far more stable when training. Yes, DPO is not technically "RL", but it's semantics for the most part. DataDreamer does support PPO training if you want, but it's so unstable, it's a less popular choice now.

Re: RLHF a LLM in <50 lines of Python

#28

Post by antonvs »

verticalscaler wrote: post #24
jerpint wrote: post #2

I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”

Yes you do. Most casuals are downright afraid of code. This messaging is meant to make the project more approachable. Kind of like everybody knows the pop-science around e = mc^2 but most are completely oblivious that it takes a bunch of whiteboards to derive it and what all that actually means. No pithy formula no way for the actual ideas to spread to the mainstream for you to somehow hear about it.

This reminds me of the advice Stephen Hawking's publisher gave him, which was that every equation he included in his book, A Brief History of Time, would cut the sales of the book in half. As a result the only equation that ended up in the book was E=mc^2.

Re: RLHF a LLM in <50 lines of Python

#29

Post by mk_stjames »

I feel the preparation and loading of the dataset has been abstracted too far away. I have no idea what type of data format I need or how it is loaded for this (it is using a pre-prepared huggingface dataset?). If I have local data how should it be loaded? What does that even look like? Is it expecting some sort of JSON?

When you get so far as to abstracting every step to loading a one-liner from huggingface, including the downloading of a prepared dataset with no example of doing the same on custom local dataset, you've abstracted too far to be useful for anyone other than the first user.

Post reply on HN