Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RLHF a LLM in <50 lines of Python

datadreamer.dev

Post reply on HN
Print view View original

61–68 of 68 posts

Re: RLHF a LLM in <50 lines of Python

#61

Post by changoplatanero »

vvrm wrote: post #51
patelajay285 wrote: post #26

Earlier quoted context omitted.

This was discussed in another comment, DPO is pretty much strictly better than RLHF + PPO, and far more stable when training. Yes, DPO is not technically "RL", but it's semantics for the most part. DataDreamer does support PPO training if you want, but it's so unstable, it's a less popular choice now.

> DPO is pretty much strictly better than RLHF + PPO Out of genuine curiosity, do you have any pointers/evidence to support this. I know that some of the industry leading research labs haven't switched over to DPO yet, in spite of the fact that DPO is significantly faster than RLHF. It might just be organizational inertia, but I do not know. I would be very happy if simpler alternatives like DPO were as good as RLHF…

I can second that. From what I’ve heard from people at leading labs, it’s not clear that dpo is worth switching to from RLHF

Re: RLHF a LLM in <50 lines of Python

#62

Post by __loam »

skelpmargyar wrote: post #45
peab wrote: post #41

Earlier quoted context omitted.

did people say the same thing when assembly code got abstracted away?

Importing a library is not abstraction any more than closing your eyes is abstracting the world to black.

The point of libraries is abstraction. You get to use the library without writing the library or knowing the underlying implementation.

Re: RLHF a LLM in <50 lines of Python

#63

Post by orblivion »

peab wrote: post #41
lopkeny12ko wrote: post #27

It's not 50 lines of code if all the real work is done by importing a library... That's like saying, I can solve any problem in 2 lines of code. I'll publish a library for it first, then: import foo; foo.do_the_thing() Magic!

did people say the same thing when assembly code got abstracted away?

There's levels of abstraction. "lines of Python" to me roughly means the standard library.

Re: RLHF a LLM in <50 lines of Python

#64

Post by sciolist »

orblivion wrote: post #63
peab wrote: post #41

Earlier quoted context omitted.

did people say the same thing when assembly code got abstracted away?

There's levels of abstraction. "lines of Python" to me roughly means the standard library.

Agreed. If it was "RLHF a LLM in <50 lines" I feel it would be clear, but "lines of Python" implies something different.

Re: RLHF a LLM in <50 lines of Python

#65

Post by v4dok »

I feel like the current meta on finetuning LLMs is random accounts at X/Twitter. Google results are littered with SEO garbage or some kind of guides that fail to work the moment you need something slightly different.

Re: RLHF a LLM in <50 lines of Python

#66

Post by rldjbpin »

it is very conflicting to see "do x in y LOC" in this field, especially when most of the workflow for different models are fragmented across non-overlapping frameworks/tooling.

to actually do something from scratch or using the author's code requires adopting something esoteric just for this purpose. for these scenarios it is nice to appreciate hf and their abstraction. but the reinventing the wheel situation is very frustrating to work with.

if you want to go beyond the demo, you have to deal with this painful reality. i hope there is more progress on this rather than making stacks of api.

Re: RLHF a LLM in <50 lines of Python

#68

Post by numeri »

Der_Einzige wrote: post #9
imjonse wrote: post #8

The first paragraphs says RLHF can be used to align models, and the seconds say here's how to do it by using DPO. These two methods are not the same, and the latter is not an instance of the former.

The latter is strictly superior to the former though. RlHF has been abandoned in the open source world.

I don't know about strictly superior. It's certainly strictly easier for people with a budget, who just need "good enough" results the first try. I don't have any evidence whatsoever, but I'd expect that enough tuning and retries can get squeeze a bit more performance out of RLHF than you can get out of DPO.
Post reply on HN