Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

github.com

Post reply on HN
Print view View original

21–30 of 90 posts

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#22

Post by falaki »

mrg3_2013 wrote: post #9

How does this compare to openai ? Curious if anyone has any anecdotes.

We don't expect this to be as good as the latest OpenAI GPT release. This is just to demonstrate that developing a conversation agent using an existing foundation model is not as hard as some may assume. Take a foundation model that is not capable of Q&A and tune it with a fairly small Q&A data and you get your in-house ChatGPT.

Disclaimer: I work at Databricks.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#24

Post by nickthegreek »

xatalytic wrote: post #2

15,000 instruction tuning records generated by Databricks employees in seven of the behavior categories outlined in the InstructGPT paper (predecessor to ChatGPT). Coincides with the release of Dolly 2.0, which is trained exclusively on this dataset and demonstrates high quality (but not state-of-the-art) instruction-following behavior. The data and models are licensed for commercial use, setting them apart from rece…

>Coincides with the release of Dolly 2.0, which is trained exclusively on this dataset and demonstrates high quality (but not state-of-the-art) instruction-following behavior.

This is not correct. It was fine-tuned with this data set, but the model itself is the 12B Eleuther AI pythia model.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#25

Post by dontupvoteme »

gamegoblin wrote: post #19
m3kw9 wrote: post #11

I’m not seeing how 15k q/a training can get you much other than the simplest things. Maybe that’s the point, get the ball rolling for people to add more training data?

What reasons do you have for believing that is true? It seems plausible to me that a general autoregressive LLM that is capable of completing text wouldn't take that much fine-tuning to shift it from "text completion" to "instruction following". After all, the raw GPT3 model can be made to follow instructions with just a few examples. Consider the prompt: What is the capital of France? Raw GPT3, not the newer instruc…

N-shot at inference-time is fundamentally different from training/fine-tuning which is inherently pre-inference-time.

Though it would be interesting to know if OpenAI has a few generic multishot inputs before the prompt.

It's all extremely cryptic what the actual context window and system prompt (assuming chatgpt even is using the same API the proles are given) is with them

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#29

Post by brianjking »

falaki wrote: post #3

This is the blog post with more details and background: https://www.databricks.com/blog/2023/04/12/dolly-first-open-... Disclosure: I work at Databricks.

Thank you and congrats to you and the team. This is fantastic

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#30

Post by mrg3_2013 »

falaki wrote: post #22
mrg3_2013 wrote: post #9

How does this compare to openai ? Curious if anyone has any anecdotes.

We don't expect this to be as good as the latest OpenAI GPT release. This is just to demonstrate that developing a conversation agent using an existing foundation model is not as hard as some may assume. Take a foundation model that is not capable of Q&A and tune it with a fairly small Q&A data and you get your in-house ChatGPT. Disclaimer: I work at Databricks.

Thanks for the feedback. The potential edge with Dolly is huge. Building a firewalled model with custom corpus is a big deal. I have been experimenting with openai and even with public data (but really limiting to the domain), yields great improvements (openai may be stale because of cut off data). I am excited to see where Dolly goes.
Post reply on HN