Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
1–10 of 90 posts
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#215,000 instruction tuning records generated by Databricks employees in seven of the behavior categories outlined in the InstructGPT paper (predecessor to ChatGPT). Coincides with the release of Dolly 2.0, which is trained exclusively on this dataset and demonstrates high quality (but not state-of-the-art) instruction-following behavior.
The data and models are licensed for commercial use, setting them apart from recent releases trained on data from OpenAI.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#3This is the blog post with more details and background: https://www.databricks.com/blog/2023/04/12/dolly-first-open-...
Disclosure: I work at Databricks.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#4This is the blog post with more details and background: https://www.databricks.com/blog/2023/04/12/dolly-first-open-... Disclosure: I work at Databricks.
We also open sourced the Dolly model itself with a license that allows commercial use.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#5Benchmarks here: https://huggingface.co/databricks/dolly-v2-12b#benchmark-met...
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#6Benchmarks here: https://huggingface.co/databricks/dolly-v2-12b#benchmark-met...
> As outlined above, these results demonstrate that dolly-v2-12b is not state of the art, and in fact underperforms dolly-v1-6b in some evaluation benchmarks. We believe this owes to the composition and size of the underlying fine tuning datasets, but a robust statement as to the sources of these variations requires further study.
Taking a moment to appreciate the integrity of the team.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#7This is the blog post with more details and background: https://www.databricks.com/blog/2023/04/12/dolly-first-open-... Disclosure: I work at Databricks.
We also open sourced the Dolly model itself with a license that allows commercial use.
can you compare your dolly offering with https://github.com/microsoft/DeepSpeedExamples/blob/master/a...
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#8Previous [flagged] discussion: https://news.ycombinator.com/item?id=35539085
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#9How does this compare to openai ? Curious if anyone has any anecdotes.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#10Benchmarks here: https://huggingface.co/databricks/dolly-v2-12b#benchmark-met...
> As outlined above, these results demonstrate that dolly-v2-12b is not state of the art, and in fact underperforms dolly-v1-6b in some evaluation benchmarks. We believe this owes to the composition and size of the underlying fine tuning datasets, but a robust statement as to the sources of these variations requires further study. Taking a moment to appreciate the integrity of the team.
Ditto, this is release early release often without necessarily meaning move fast and break things. Other teams can do the equivalent of Alpaca to Llama and we can all learn for the next round.