I’m not seeing how 15k q/a training can get you much other than the simplest things. Maybe that’s the point, get the ball rolling for people to add more training data?
Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
11–20 of 90 posts
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#12This is the blog post with more details and background: https://www.databricks.com/blog/2023/04/12/dolly-first-open-... Disclosure: I work at Databricks.
We also open sourced the Dolly model itself with a license that allows commercial use.
How hard would it be to get dolly running on llama.cpp?
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#13Earlier quoted context omitted.
> As outlined above, these results demonstrate that dolly-v2-12b is not state of the art, and in fact underperforms dolly-v1-6b in some evaluation benchmarks. We believe this owes to the composition and size of the underlying fine tuning datasets, but a robust statement as to the sources of these variations requires further study. Taking a moment to appreciate the integrity of the team.
Ditto, this is release early release often without necessarily meaning move fast and break things. Other teams can do the equivalent of Alpaca to Llama and we can all learn for the next round.
One of the creators here - yeah, the thing we have our eyes on is the vector not the point.
It’s astounding how adaptable these open models are, even with just a quarter of the Alpaca data. We’re a team of machine learning engineers and hackers, not an AI science lab, but that’s kind of the point frankly - this whole exercise appears to be far easier that it might at first seem.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#14This is the blog post with more details and background: https://www.databricks.com/blog/2023/04/12/dolly-first-open-... Disclosure: I work at Databricks.
Thank you, thank you, thank you!
If possible, could you share how Dolly v2 compares to RWKV-4 14B ctx 8019?
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#15I don't think these upvotes are organic.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#16I don't think these upvotes are organic.
Why do you say that?
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#17I’m not seeing how 15k q/a training can get you much other than the simplest things. Maybe that’s the point, get the ball rolling for people to add more training data?
It's used for fine tuning a pre-trained model. This takes an LLM that is already capable of emulating lots of different kinds of personalities, and narrows it down to act more like the examples. Since the heavy lifting has already been done, 15k examples of a chatbot following instructions they way you want has a significant effect.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#18Great to see more releases under open licenses!
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#19I’m not seeing how 15k q/a training can get you much other than the simplest things. Maybe that’s the point, get the ball rolling for people to add more training data?
What reasons do you have for believing that is true?
It seems plausible to me that a general autoregressive LLM that is capable of completing text wouldn't take that much fine-tuning to shift it from "text completion" to "instruction following".
After all, the raw GPT3 model can be made to follow instructions with just a few examples.
Consider the prompt:
What is the capital of France?
What is the capital of France? If a student answers with a word,
she is asked to identify the word. She is not asked whether the
capital of France is Paris. On the other hand, if the student
answers by pointing to a map, she is asked to identify the capital
of France. She is not asked whether it is Paris.
But if you give it a few examples, it happily gets into instruction following mode:
The following is a transcript between a human and a helpful
AI assistant who answers questions and obeys commands.
Human: How many eggs are in a dozen?
AI: 12
Human: Say "hello" 3 times
AI: hello hello hello
Human: What is the capital of France?
AI:
If you can get decent instruction/question following behavior out of a 2-shot example prompt, why do you think 15k is small for this?
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#20Here's a link to open up and explore that training data in Datasette Lite: https://lite.datasette.io/?json=https://github.com/databrick...