Show HN: Implementation of the "Self-Rewarding Language Models" Paper by MetaAI
1–7 of 7 posts
Re: Show HN: Implementation of the "Self-Rewarding Language Models" Paper by MetaAI
#2Hey all,
After reading the Self-Rewarding Language Models paper by the team at Meta, it felt very approachable and reproducible, so we spent some time implementing it.
The scripts provided take any base model and put it in a loop of:
1) Supervised fine-tuning on an initial dataset
2) Generating new prompts using the SFT
3) Generating N responses per prompt
4) Scoring the generated responses 1-5
5) Running DPO on the rewards from the model itself.
We've run it through one loop starting with a Mistral-7b base model and the results are pretty encouraging so far.
Feel free to check it out or run it for yourself and let us know what you think:
Re: Show HN: Implementation of the "Self-Rewarding Language Models" Paper by MetaAI
#3Hey all, After reading the Self-Rewarding Language Models paper by the team at Meta, it felt very approachable and reproducible, so we spent some time implementing it. The scripts provided take any base model and put it in a loop of: 1) Supervised fine-tuning on an initial dataset 2) Generating new prompts using the SFT 3) Generating N responses per prompt 4) Scoring the generated responses 1-5 5) Running DPO on the…
What did you use for specs? 32G of memory is enough?
Re: Show HN: Implementation of the "Self-Rewarding Language Models" Paper by MetaAI
#4[deleted]
Re: Show HN: Implementation of the "Self-Rewarding Language Models" Paper by MetaAI
#5Hey all, After reading the Self-Rewarding Language Models paper by the team at Meta, it felt very approachable and reproducible, so we spent some time implementing it. The scripts provided take any base model and put it in a loop of: 1) Supervised fine-tuning on an initial dataset 2) Generating new prompts using the SFT 3) Generating N responses per prompt 4) Scoring the generated responses 1-5 5) Running DPO on the…
What's the goal of a self rewarding llm?
Re: Show HN: Implementation of the "Self-Rewarding Language Models" Paper by MetaAI
#6Hey all, After reading the Self-Rewarding Language Models paper by the team at Meta, it felt very approachable and reproducible, so we spent some time implementing it. The scripts provided take any base model and put it in a loop of: 1) Supervised fine-tuning on an initial dataset 2) Generating new prompts using the SFT 3) Generating N responses per prompt 4) Scoring the generated responses 1-5 5) Running DPO on the…
What's the goal of a self rewarding llm?
The goal is to iteratively create training data and add it to its own training set. The LLM acts as its own judge and scores its own responses to decide if it should add the data. It’s expensive to have a human in the loop labeling preferences, so the folks at Meta showed you can have a clever prompt and fine tune the model to judge its own responses.
Re: Show HN: Implementation of the "Self-Rewarding Language Models" Paper by MetaAI
#7Hey all, After reading the Self-Rewarding Language Models paper by the team at Meta, it felt very approachable and reproducible, so we spent some time implementing it. The scripts provided take any base model and put it in a loop of: 1) Supervised fine-tuning on an initial dataset 2) Generating new prompts using the SFT 3) Generating N responses per prompt 4) Scoring the generated responses 1-5 5) Running DPO on the…
What did you use for specs? 32G of memory is enough?
We used an A10 with 24GB of VRAM, this was enough for PEFT on Mistral-7B