Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

github.com

Post reply on HN
Print view View original

51–60 of 90 posts

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#51

Post by mrtranscendence »

rnk wrote: post #41
simonw wrote: post #21

I got this model working on a GPU instance, notes here: https://til.simonwillison.net/llms/dolly-2 Anyone managed to run it on an M1/M2 Mac yet?

How much ram is likely needed on an apple arm for models like this? And for general use, 64, 96, 128? Trying to decide how large I should go for a new laptop.

I very recently purchased a MacBook Pro (M1 Max) with 64GB of ram. I haven't experimented that much, but I was able to run inference using the 65B parameter Llama model with quantized weights at a speed that was reasonably usable (maybe a touch slower than ChatGPT with GPT-4).

I haven't attempted to use the 65B model with non-quantized weights, but the smaller models work that way, if slowly. With 96GB of ram -- the upper limit of a MacBook Pro -- you might be able to use even larger models, but I think you'd hit the limits of useful performance before that point.

I should note that it can be a bit tricky getting things to work using the Mac's GPU. I couldn't get Dolly 6B to run on my work MBP, which theoretically should have enough ram, though I still want to try it on my personal laptop.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#53

Post by Szpadel »

falaki wrote: post #22
mrg3_2013 wrote: post #9

How does this compare to openai ? Curious if anyone has any anecdotes.

We don't expect this to be as good as the latest OpenAI GPT release. This is just to demonstrate that developing a conversation agent using an existing foundation model is not as hard as some may assume. Take a foundation model that is not capable of Q&A and tune it with a fairly small Q&A data and you get your in-house ChatGPT. Disclaimer: I work at Databricks.

you are referring to the dolly model? I think the training set could achieve similar performance if we would fine tune similarly sized model

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#54

Post by rnk »

Szpadel wrote: post #48
rnk wrote: post #41

Earlier quoted context omitted.

How much ram is likely needed on an apple arm for models like this? And for general use, 64, 96, 128? Trying to decide how large I should go for a new laptop.

AFAIK current models can run even with 64GB, but I would assume that we will very likely have bigger models very soon so I guess the answer is as much as you can afford

The next question is m1 or m2, and the impact of the various number of gpu units between pro, max, ultra skews. I'm really tempted to buy a "refurbished m1 studio" with 128gb because I think the ram is the key. Have not seen any benchmarks with diff # of gpus/aka diff skews.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#55

Post by thewataccount »

itake wrote: post #52
mydpy wrote: post #5

Benchmarks here: https://huggingface.co/databricks/dolly-v2-12b#benchmark-met...

Why are they not doing metrics against GPT-3.5 and GPT-4? My understanding is Dolly performs significantly worse.

I haven't played with the model just yet - but just eye balling it's performance it's significantly worse. I'm surprised they don't have Pythia on there as that's what they're based on from my understanding.

At their performance level it's the most important to compare to GPT-neoX, and I do appreciate they aren't making the "95% of GPT4" claims that some fine-tuned llama models are.

EDIT: For databricks people: I'd love to see this compared with Pythia, LLaMa, Alpaca, and vicuna/gpt4all if possible.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#56

Post by rnk »

mrtranscendence wrote: post #51
rnk wrote: post #41

Earlier quoted context omitted.

How much ram is likely needed on an apple arm for models like this? And for general use, 64, 96, 128? Trying to decide how large I should go for a new laptop.

I very recently purchased a MacBook Pro (M1 Max) with 64GB of ram. I haven't experimented that much, but I was able to run inference using the 65B parameter Llama model with quantized weights at a speed that was reasonably usable (maybe a touch slower than ChatGPT with GPT-4). I haven't attempted to use the 65B model with non-quantized weights, but the smaller models work that way, if slowly. With 96GB of ram -- the…

I see refurbished m1 2tb/128gb for $4700, looks like similar price for an m2 with same storage/ram with my corp discount (20cpu/48gpu). This is a tough decision.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#57

Post by thewataccount »

falaki wrote: post #3

This is the blog post with more details and background: https://www.databricks.com/blog/2023/04/12/dolly-first-open-... Disclosure: I work at Databricks.

Would you consider adding Pythia12B, LLaMa and Alpaca since that's what you're directly compared against/based on?

GPT3.5/GPT4 is what everyone would also love to see but I understand you're performance is inline with GPT-neoX.

Vicuna/GPT4all would be intersting but IMO are less important.

RWKV would be interesting because it's a completely different model from the transformers.

EDIT: Also thanks for the opensource contributions! Highly appreciated!

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#58

Post by theGnuMe »

mrtranscendence wrote: post #39
mrg3_2013 wrote: post #30

Earlier quoted context omitted.

Thanks for the feedback. The potential edge with Dolly is huge. Building a firewalled model with custom corpus is a big deal. I have been experimenting with openai and even with public data (but really limiting to the domain), yields great improvements (openai may be stale because of cut off data). I am excited to see where Dolly goes.

Dolly appears to fundamentally be a tech demo advertising how you can use Databricks for compute. I honestly wouldn't expect them to take it that much further, particularly in the context of larger models that would be significantly more expensive to fine-tune. But I'm happy to be proven wrong.

I imagine they will sell fine tuning as a service to Databricks customers. If I put all my data into their lake I too can get my own custom ChatGPT. That's compelling.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#59

Post by jarek83 »

simonw wrote: post #20

Here's a link to open up and explore that training data in Datasette Lite: https://lite.datasette.io/?json=https://github.com/databrick...

Can someone help me to understand why categories for these two differ?

row #51 "Think of some family rules to promote a healthy family relationship" - brainstorsming [1]

row #68 "What is the future for human?" - general_qa [2]

In nature they both are brainstorming to me - does the question mark is what assigned the #68 as _qa?

[1] https://lite.datasette.io/?json=https://github.com/databrick...

[2] https://lite.datasette.io/?json=https://github.com/databrick...

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#60

Post by whimsicalism »

m3kw9 wrote: post #11

I’m not seeing how 15k q/a training can get you much other than the simplest things. Maybe that’s the point, get the ball rolling for people to add more training data?

Read about RLHF, i think you are misunderstanding what this will be used for.
Post reply on HN