Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RLHF a LLM in <50 lines of Python

datadreamer.dev

Post reply on HN
Print view View original

31–40 of 68 posts

Re: RLHF a LLM in <50 lines of Python

#31

Post by patelajay285 »

mk_stjames wrote: post #29

I feel the preparation and loading of the dataset has been abstracted too far away. I have no idea what type of data format I need or how it is loaded for this (it is using a pre-prepared huggingface dataset?). If I have local data how should it be loaded? What does that even look like? Is it expecting some sort of JSON? When you get so far as to abstracting every step to loading a one-liner from huggingface, includi…

Thanks for the question. This is built for ML researchers, so in examples we use the defacto source for datasets researchers often use, HF Hub.

However, there is a lot of documentation on the site to help guide users. This documentation page shows you can load in data via local datasets as well. For example, JSON, CSV, text files, a local HF Dataset folder, or even from a Python `dict` or `list`:

https://datadreamer.dev/docs/latest/datadreamer.steps.html#t...

We'll definitely keep improving documentation, guides, and examples. We have a lot of it already, and more to come! This has only recently become a public project :)

If anyone has any questions on using it, feel free to email me directly (email on the site and HN bio) for help in the meantime.

Re: RLHF a LLM in <50 lines of Python

#32

Post by antonvs »

lopkeny12ko wrote: post #27

It's not 50 lines of code if all the real work is done by importing a library... That's like saying, I can solve any problem in 2 lines of code. I'll publish a library for it first, then: import foo; foo.do_the_thing() Magic!

Software developers hate this one simple trick!

Re: RLHF a LLM in <50 lines of Python

#33

Post by MrYellowP »

I don't prefer aligned models and I'm a human. It's not okay to claim that that's what humans prefer. There might be a subset of humans who can't handle words, but they're not even remotely in the majority.

Algined models are dumber, treat everyone like they're stupid immature idiots who can't handle words and they're a wannabe moral authority.

Re: RLHF a LLM in <50 lines of Python

#34

Post by [deleted] »

mk_stjames wrote: post #29

I feel the preparation and loading of the dataset has been abstracted too far away. I have no idea what type of data format I need or how it is loaded for this (it is using a pre-prepared huggingface dataset?). If I have local data how should it be loaded? What does that even look like? Is it expecting some sort of JSON? When you get so far as to abstracting every step to loading a one-liner from huggingface, includi…

[deleted]

Re: RLHF a LLM in <50 lines of Python

#35

Post by antonvs »

patelajay285 wrote: post #26
vvrm wrote: post #25

Earlier quoted context omitted.

Another problem with the title: the article is about DPO, which doesn’t do reinforcement learning. So not RLHF. I guess RLHF has more of a name recognition than DPO.

This was discussed in another comment, DPO is pretty much strictly better than RLHF + PPO, and far more stable when training. Yes, DPO is not technically "RL", but it's semantics for the most part. DataDreamer does support PPO training if you want, but it's so unstable, it's a less popular choice now.

In the DPO paper linked from the OP page, DPO is described as "a simple RL-free algorithm for training language models from preferences." So as you say, "not technically RL."

Given that, shouldn't the first sentence on the linked page end with "...in a process known as DPO (...)" ? Ditto for the title.

It sounds like you're saying that the terms RL and RLHF should subsume DPO because they both solve the same problem, with similar results. But they're different techniques, and there are established terms for both of them.

Re: RLHF a LLM in <50 lines of Python

#36

Post by patelajay285 »

antonvs wrote: post #35
patelajay285 wrote: post #26

Earlier quoted context omitted.

This was discussed in another comment, DPO is pretty much strictly better than RLHF + PPO, and far more stable when training. Yes, DPO is not technically "RL", but it's semantics for the most part. DataDreamer does support PPO training if you want, but it's so unstable, it's a less popular choice now.

In the DPO paper linked from the OP page, DPO is described as "a simple RL-free algorithm for training language models from preferences." So as you say, "not technically RL." Given that, shouldn't the first sentence on the linked page end with "...in a process known as DPO (...)" ? Ditto for the title. It sounds like you're saying that the terms RL and RLHF should subsume DPO because they both solve the same problem,…

I think the discussion in the other comment thread discusses this well. They are different techniques, but the line between RL & SL is quite fuzzy. The DPO authors advertise this as a "non-RL" technique to precisely get away from the reputation of unstable training RL has, but they also say and treat the language model as an (implicit) reward model, similar to PPO. The point is well taken though, I will update this page to clarify the differences to avoid confusion.

Re: RLHF a LLM in <50 lines of Python

#37

Post by mk_stjames »

patelajay285 wrote: post #31
mk_stjames wrote: post #29

I feel the preparation and loading of the dataset has been abstracted too far away. I have no idea what type of data format I need or how it is loaded for this (it is using a pre-prepared huggingface dataset?). If I have local data how should it be loaded? What does that even look like? Is it expecting some sort of JSON? When you get so far as to abstracting every step to loading a one-liner from huggingface, includi…

Thanks for the question. This is built for ML researchers, so in examples we use the defacto source for datasets researchers often use, HF Hub. However, there is a lot of documentation on the site to help guide users. This documentation page shows you can load in data via local datasets as well. For example, JSON, CSV, text files, a local HF Dataset folder, or even from a Python `dict` or `list`: https://datadreamer.…

I did glance at the docs first before commenting but I was looking in 'datasets' to try and understand importing a potential CSV/JOSN etc and all I saw was verbage on accessing the output.

I would not have guessed that the base input data processing would have been filed under 'steps'. But now I kinda see how you are working, but I admit I'm not the target audience.

If you want this to really take off for people outside of a very, very specific class of researchers... setup an example on your landing page that calls to a local JSON of user prompts/answers/rejects finetuning a llama model with your datadreamer.steps.JSONDataSource into the loader. Or, a txt file with the system/user/assistant prompts tagged and examples given. Yes, your 'lines of code' for your frontpage example may grow a bit!

Maybe there are a lot of 'ML researchers' that are used to the type of super-abstract OOP API, load-it-from-huggingface-scheme-people you are targeting but also know that there are a ton that aren't.

Re: RLHF a LLM in <50 lines of Python

#38

Post by patelajay285 »

mk_stjames wrote: post #37
patelajay285 wrote: post #31

Earlier quoted context omitted.

Thanks for the question. This is built for ML researchers, so in examples we use the defacto source for datasets researchers often use, HF Hub. However, there is a lot of documentation on the site to help guide users. This documentation page shows you can load in data via local datasets as well. For example, JSON, CSV, text files, a local HF Dataset folder, or even from a Python `dict` or `list`: https://datadreamer.…

I did glance at the docs first before commenting but I was looking in 'datasets' to try and understand importing a potential CSV/JOSN etc and all I saw was verbage on accessing the output. I would not have guessed that the base input data processing would have been filed under 'steps'. But now I kinda see how you are working, but I admit I'm not the target audience. If you want this to really take off for people outs…

That's totally fair and good feedback, it's hard to support everyone's use cases simultaneously, but from my own research and other researchers we collaborate with, this solves and streamlines the right set of problems, but we want to make this as broadly useful as possible. Always happy to chat more / provide support if you would like, feel free to reach out if you want to try it and run into any sharp edges I could help make easier.
Post reply on HN