I got this model working on a GPU instance, notes here: https://til.simonwillison.net/llms/dolly-2 Anyone managed to run it on an M1/M2 Mac yet?
I'm sure we'll see this by the end of the day or two.
Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
31–40 of 90 posts
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#32Happy to see this type of work that is truly open source and commercially usable. Is this the entire corpus or a subset? Do you intend to release any new iterations?
I've been thinking of starting similar efforts at another BigCorp by hosting a UL2 or GPT-J instance.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#33Earlier quoted context omitted.
What reasons do you have for believing that is true? It seems plausible to me that a general autoregressive LLM that is capable of completing text wouldn't take that much fine-tuning to shift it from "text completion" to "instruction following". After all, the raw GPT3 model can be made to follow instructions with just a few examples. Consider the prompt: What is the capital of France? Raw GPT3, not the newer instruc…
N-shot at inference-time is fundamentally different from training/fine-tuning which is inherently pre-inference-time. Though it would be interesting to know if OpenAI has a few generic multishot inputs before the prompt. It's all extremely cryptic what the actual context window and system prompt (assuming chatgpt even is using the same API the proles are given) is with them
The claim is not that they are fundamentally different or similar, the claim is that one doesn't need that much data to get instruction-following behavior from a raw autoregressive LLM. K-shot prompting shows that the capability to follow instructions is present in the model. It's just a matter of using fine-tuning to keep the model in that frame all the time without a K-shot prompt.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#34I got this model working on a GPU instance, notes here: https://til.simonwillison.net/llms/dolly-2 Anyone managed to run it on an M1/M2 Mac yet?
Not on M1/M2 yet, but my response time seems pretty fast on Tesla V100-SXM2-16GB
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#35Earlier quoted context omitted.
Why do you say that?
probably based on the situation 3hrs ago - https://news.ycombinator.com/item?id=35539085
There's a big difference between employees who got excited to see their work on hacker news and upvoted it and premeditated shill / astroturf campaign. We should pretty much assume that a San Fransisco based company is going to have significant readership / membership here.
One can easily see how a message over a company communicator could result in a surge of upvotes.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#3615,000 instruction tuning records generated by Databricks employees in seven of the behavior categories outlined in the InstructGPT paper (predecessor to ChatGPT). Coincides with the release of Dolly 2.0, which is trained exclusively on this dataset and demonstrates high quality (but not state-of-the-art) instruction-following behavior. The data and models are licensed for commercial use, setting them apart from rece…
>Coincides with the release of Dolly 2.0, which is trained exclusively on this dataset and demonstrates high quality (but not state-of-the-art) instruction-following behavior. This is not correct. It was fine-tuned with this data set, but the model itself is the 12B Eleuther AI pythia model.
There are two, a 6B parameter model fine-tuned on GPT-J and a 12B parameter model fine-tuned on Pythia.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#37I got this model working on a GPU instance, notes here: https://til.simonwillison.net/llms/dolly-2 Anyone managed to run it on an M1/M2 Mac yet?
I attempted using the Transformers library but failed. Not sure, might be a VRAM issue; I'm going to try on my far beefier personal MacBook Pro later tonight.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#38Happy to see this type of work that is truly open source and commercially usable. Is this the entire corpus or a subset? Do you intend to release any new iterations? I've been thinking of starting similar efforts at another BigCorp by hosting a UL2 or GPT-J instance.
15k is the entire corpus we have right now. Hopefully others can join up in releasing additional samples that can be merged in over time.
We'll definitely keep iterating on Dolly and releasing everything openly.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#39Earlier quoted context omitted.
We don't expect this to be as good as the latest OpenAI GPT release. This is just to demonstrate that developing a conversation agent using an existing foundation model is not as hard as some may assume. Take a foundation model that is not capable of Q&A and tune it with a fairly small Q&A data and you get your in-house ChatGPT. Disclaimer: I work at Databricks.
Thanks for the feedback. The potential edge with Dolly is huge. Building a firewalled model with custom corpus is a big deal. I have been experimenting with openai and even with public data (but really limiting to the domain), yields great improvements (openai may be stale because of cut off data). I am excited to see where Dolly goes.
Dolly appears to fundamentally be a tech demo advertising how you can use Databricks for compute. I honestly wouldn't expect them to take it that much further, particularly in the context of larger models that would be significantly more expensive to fine-tune. But I'm happy to be proven wrong.
Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
#40Earlier quoted context omitted.
probably based on the situation 3hrs ago - https://news.ycombinator.com/item?id=35539085
There's a big difference between employees who got excited to see their work on hacker news and upvoted it and premeditated shill / astroturf campaign. We should pretty much assume that a San Fransisco based company is going to have significant readership / membership here. One can easily see how a message over a company communicator could result in a surge of upvotes.
Agreed. I was just providing the context that the user asked for.