11–20 of 68 posts
The first paragraphs says RLHF can be used to align models, and the seconds say here's how to do it by using DPO. These two methods are not the same, and the latter is not an instance of the former.
The latter is strictly superior to the former though. RlHF has been abandoned in the open source world.
The first paragraphs says RLHF can be used to align models, and the seconds say here's how to do it by using DPO. These two methods are not the same, and the latter is not an instance of the former.
The latter is strictly superior to the former though. RlHF has been abandoned in the open source world.
Hi everyone, there are no easy tools for synthetic data generation or training and aligning LLMs simply in Python. Most of the stuff out there are messy adhoc scripts. DataDreamer is an open source Python package with a nice API from the University of Pennsylvania that does all this that we’re actively developing. Will be here to answer questions. https://github.com/datadreamer-dev/DataDreamer
Hi everyone, there are no easy tools for synthetic data generation or training and aligning LLMs simply in Python. Most of the stuff out there are messy adhoc scripts. DataDreamer is an open source Python package with a nice API from the University of Pennsylvania that does all this that we’re actively developing. Will be here to answer questions. https://github.com/datadreamer-dev/DataDreamer
The API looks nice, congratulations. Will experiment with it. One small silly question: why did you choose to specify the dependencies inside the src dir with the requirements format - rather than inside the pyproject?
I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”
I always appreciate these projects because I just dive into the code itself and copy out what I need once the wrapper becomes too much of a burden.
I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”
I always appreciate these projects because I just dive into the code itself and copy out what I need once the wrapper becomes too much of a burden.
They're saying why does it matter if it's 50 vs 60 or even 100. It's a wrapper, which should be less lines. That's the whole point. Abstracting things even further and making assumptions.
Of course you can use them. Of course you can remove them after and use the underlying code. But the LOC shouldn't be the important part of it
Earlier quoted context omitted.
The latter is strictly superior to the former though. RlHF has been abandoned in the open source world.
Yep, DPO is not technically “RL” and implicitly uses the LLM itself as a reward model, but training with DPO is far more stable for that reason.
I'm not a fan of the RL/SL dichotomy, because the line gets so foggy. If you squint, every loss is a negative reward, and every policy improvement a supervised target.
Still, what the code does isn't what is described in the paper that the page links to.
Earlier quoted context omitted.
The latter is strictly superior to the former though. RlHF has been abandoned in the open source world.
Yep, DPO is not technically “RL” and implicitly uses the LLM itself as a reward model, but training with DPO is far more stable for that reason.
I don’t understand the obsession of LOC for wrappers - it’s the whole point of a wrapper. It makes it much easier for the user at the expense of making it less hackable Title should be instead “Library for low-code RLHF in python”