Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RLHF a LLM in <50 lines of Python

datadreamer.dev

Post reply on HN
Print view View original

41–50 of 68 posts

Re: RLHF a LLM in <50 lines of Python

#41

Post by peab »

lopkeny12ko wrote: post #27

It's not 50 lines of code if all the real work is done by importing a library... That's like saying, I can solve any problem in 2 lines of code. I'll publish a library for it first, then: import foo; foo.do_the_thing() Magic!

did people say the same thing when assembly code got abstracted away?

Re: RLHF a LLM in <50 lines of Python

#44

Post by theptip »

Interested if local RLHF is actually viable; can you get meaningful steering from 1k feedback points on a narrow task? I feel that annotation count is achievable with a single dedicated annotator making a few comments per minute (though tedious), 10k would be a week of work so achievable for a very dedicated hobbyist, and 100k seems out of reach for a hobby project.

Say for simple conversation usecases (eg customer support for a specific product, interactive fiction, things like that without deep technical knowledge).

I was also wondering if it’s possible to do such RLHF for SD running locally.

Re: RLHF a LLM in <50 lines of Python

#45

Post by skelpmargyar »

peab wrote: post #41
lopkeny12ko wrote: post #27

It's not 50 lines of code if all the real work is done by importing a library... That's like saying, I can solve any problem in 2 lines of code. I'll publish a library for it first, then: import foo; foo.do_the_thing() Magic!

did people say the same thing when assembly code got abstracted away?

Importing a library is not abstraction any more than closing your eyes is abstracting the world to black.

Re: RLHF a LLM in <50 lines of Python

#48

Post by patelajay285 »

aethelyon wrote: post #47

This is cool, but the data collection is the hard part, right?

Yes it is :), but the library is also a synthetic data generation library, so for example you can create the data for DPO fully synthetically, check out the self-rewarding LLMs example:

https://datadreamer.dev/docs/latest/pages/get_started/quick_...

Re: RLHF a LLM in <50 lines of Python

#49

Post by patelajay285 »

ilaksh wrote: post #46

How do you normally do DPO? Is that built in to PyTorch or something? Theoretically the hard part is collecting the examples with rejections etc.

Collecting data is hard, but the library is also a synthetic data generation library, so for example you can create the data for DPO fully synthetically, check out the self-rewarding LLMs example: https://datadreamer.dev/docs/latest/pages/get_started/quick_...

Re: RLHF a LLM in <50 lines of Python

#50

Post by sillysaurusx »

patelajay285 wrote: post #48
aethelyon wrote: post #47

This is cool, but the data collection is the hard part, right?

Yes it is :), but the library is also a synthetic data generation library, so for example you can create the data for DPO fully synthetically, check out the self-rewarding LLMs example: https://datadreamer.dev/docs/latest/pages/get_started/quick_...

I’m extremely skeptical of this approach. Until proven otherwise, with a model that users actually find useful, I don’t think this can work.

It would be nice. But I’ve seen too many nice ideas completely fall apart in practice to accept this without some justification. Even if there are papers on the topic, and those papers show that the models rank highly according to some eval metrics, the only metric that truly matters is "the user likes the model and it solves their problems."

By the way, on a separate topic, the 90/10 dataset split that you do in all of your examples turns out to be fraught with peril in practice. The issue is that the validation dataset quality turns out to be crucial, and randomly yeeting 10% of your data into the validation dataset without manual review is a recipe for problems.

Post reply on HN