RLHF a LLM in <50 lines of Python
datadreamer.dev
RLHF a LLM in <50 lines of Python
1–10 of 68 posts
Re: RLHF a LLM in <50 lines of Python
#2I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable
Title should be instead “Library for low-code RLHF in python”
Re: RLHF a LLM in <50 lines of Python
#3Hi everyone, there are no easy tools for synthetic data generation or training and aligning LLMs simply in Python. Most of the stuff out there are messy adhoc scripts.
DataDreamer is an open source Python package with a nice API from the University of Pennsylvania that does all this that we’re actively developing. Will be here to answer questions.
Re: RLHF a LLM in <50 lines of Python
#4Very cool, but I can't help but feel like titles that reference low-LOC are a bit clickbait-y when nearly all the heavy lifting is done by imported libraries.
Re: RLHF a LLM in <50 lines of Python
#5I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”
This is developed for researchers, so I assure it’s very hackable and configurable. ;-) but appreciate the feedback on the title!
Re: RLHF a LLM in <50 lines of Python
#6Very cool, but I can't help but feel like titles that reference low-LOC are a bit clickbait-y when nearly all the heavy lifting is done by imported libraries.
Appreciate the feedback on the title, this is developed for ML researchers, so I assure there is a lot it’s doing under the hood to make this process easier (for example introducing automatic caching and resumability).
However, we also tried to simplify the API and have sensible defaults to make it usable for anyone / make ML research code cleaner :)
Re: RLHF a LLM in <50 lines of Python
#7I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”
I always appreciate these projects because I just dive into the code itself and copy out what I need once the wrapper becomes too much of a burden.
Re: RLHF a LLM in <50 lines of Python
#8The first paragraphs says RLHF can be used to align models, and the seconds say here's how to do it by using DPO. These two methods are not the same, and the latter is not an instance of the former.
Re: RLHF a LLM in <50 lines of Python
#9The first paragraphs says RLHF can be used to align models, and the seconds say here's how to do it by using DPO. These two methods are not the same, and the latter is not an instance of the former.
The latter is strictly superior to the former though. RlHF has been abandoned in the open source world.
Re: RLHF a LLM in <50 lines of Python
#10The first paragraphs says RLHF can be used to align models, and the seconds say here's how to do it by using DPO. These two methods are not the same, and the latter is not an instance of the former.
Fair, DPO is considered a fairly well established technique now that is far more stable in training than PPO, but also helps align LLMs from human feedback. The package also helps do PPO, so you can do traditional RLHF, but figured more people would be interested in seeing a DPO example, given how unstable PPO is.