Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

github.com

Post reply on HN
Print view View original

71–80 of 90 posts

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#71

Post by rnk »

anentropic wrote: post #70
rnk wrote: post #54

Earlier quoted context omitted.

The next question is m1 or m2, and the impact of the various number of gpu units between pro, max, ultra skews. I'm really tempted to buy a "refurbished m1 studio" with 128gb because I think the ram is the key. Have not seen any benchmarks with diff # of gpus/aka diff skews.

I saw this: https://github.com/jankais3r/LLaMA_MPS it runs slightly slower on the GPU than under llama.cpp but uses much less power doing so I would guess the slowness is due to immaturity of the PyTorch MPS backend, the asitop graphs show it doing a bunch of cpu along with the gpu, so it might be inefficiently falling back to cpu for some ops and swapping layers back and forth (I have no idea, just guessing)

Hey, thanks so much. That solidifies the case for 128gb mac studio. Apple could be selling a bunch of these things with these high ram capabilities.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#72

Post by esafak »

whimsicalism wrote: post #60
m3kw9 wrote: post #11

I’m not seeing how 15k q/a training can get you much other than the simplest things. Maybe that’s the point, get the ball rolling for people to add more training data?

Read about RLHF, i think you are misunderstanding what this will be used for.

A specific reference would help readers.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#73

Post by itamarcode »

"dolly-v2-12b is not a state-of-the-art generative language model and, though quantitative benchmarking is ongoing, is not designed to perform competitively with more modern model architectures or models subject to larger pretraining corpuses." from: https://huggingface.co/databricks/dolly-v2-12b

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#74

Post by whimsicalism »

esafak wrote: post #72
whimsicalism wrote: post #60

Earlier quoted context omitted.

Read about RLHF, i think you are misunderstanding what this will be used for.

A specific reference would help readers.

good point! https://huggingface.co/blog/rlhf :)

i think the resources out there so far are not great yet

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#75

Post by ankitmathur »

oidar wrote: post #12
falaki wrote: post #4

Earlier quoted context omitted.

We also open sourced the Dolly model itself with a license that allows commercial use.

How hard would it be to get dolly running on llama.cpp?

Hey there! I worked on Dolly, and I work on Model Serving at Databricks. DollyV1 is GPT-J-based, so it'll run easily on llama.cpp. DollyV2 is Pythia-based, which is built with the GPT-NeoX library

GPT-NeoX is not that different than GPT-J (it also has the rotary embeddings, which llama.cpp supports for GPT-J). I would imagine it's not too heavy of a lift to add NeoX architecture support

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#76

Post by cma »

anentropic wrote: post #68
mrtranscendence wrote: post #36

Earlier quoted context omitted.

There are two, a 6B parameter model fine-tuned on GPT-J and a 12B parameter model fine-tuned on Pythia.

the GPT-J-6B one is Dolly 1.0, previously released Dolly 2.0 is Pythia-12B fine-tuned on this new dataset on their hugging face page [1] they admit the performance may not be much or any better than the original model (I am guessing this may be a weakness of Pythia-12B, which was intended for model-training research rather than best results) the main point of Dolly 2.0 is the new dataset is unencumbered legally [2] w…

I think there's probably nothing wrong with training on others' ChatGPT transcripts posted on the open web. OpenAI trains on source-available projects with non-commercial terms, so their lawyers have already been over a similar case and decided it should be fine.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#77

Post by ankitmathur »

thewataccount wrote: post #55
itake wrote: post #52

Earlier quoted context omitted.

Why are they not doing metrics against GPT-3.5 and GPT-4? My understanding is Dolly performs significantly worse.

I haven't played with the model just yet - but just eye balling it's performance it's significantly worse. I'm surprised they don't have Pythia on there as that's what they're based on from my understanding. At their performance level it's the most important to compare to GPT-neoX, and I do appreciate they aren't making the "95% of GPT4" claims that some fine-tuned llama models are. EDIT: For databricks people: I'd l…

Out of curiosity: what's an example of a metric that you would use to evaluate the ability of the model? For example, just looking qualitatively, asking a prompt like "How do I tie a tie?" to Pythia produces content that isn't even reasonably responding to that. And yet many benchmarks have no problem with that

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#78

Post by Daegalus »

zamnos wrote: post #62
Daegalus wrote: post #50

Earlier quoted context omitted.

Im not an expert, and I don't have nvidia, but I assume you need to setup CUDA and install the CUDA pytorch stuff? Most docs Ive read on setting up finetuners and inference require some extra stuff. Taking some LORA fine tuners, they include instructions like this: conda create -n llm-finetuner python=3.10 conda activate llm-finetuner conda install -y cuda -c nvidia/label/cuda-11.7.0 conda install -y pytorch=2 pytorc…

The error message implies that the compiled default libraries on the M1 don't support the model format, even though it works fine in Paperspace. The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. Traceback (most recent call last): File "/Users/fragmede/projects/llm/dolly/foo.py", line 5, in instruct_pipeline = pipeline( ^^^^^^^^^ File "/Library/Frameworks/Python.fr…

I was referring to his TIL post about setting it up on paperspace, not about apple hardware.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#79

Post by ankitmathur »

gpm wrote: post #64
jarek83 wrote: post #59

Earlier quoted context omitted.

Can someone help me to understand why categories for these two differ? row #51 "Think of some family rules to promote a healthy family relationship" - brainstorsming [1] row #68 "What is the future for human?" - general_qa [2] In nature they both are brainstorming to me - does the question mark is what assigned the #68 as _qa? [1] https://lite.datasette.io/?json=https://github.com/databrick... [2] https://lite.datase…

The labelling doesn't seem to be entirely consistent to me, but I think the idea is that 51 is inviting you to brainstorm, while 68 is asking a question that just happens to be open ended.

Hey! Worked on this here at Databricks: the blog post goes into the dataset collection design a bit (https://www.databricks.com/blog/2023/04/12/dolly-first-open-...). In summary, you're right - brainstorming and GeneralQA will have overlap because the taxonomy naturally has some overlap

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#80

Post by zamnos »

Daegalus wrote: post #78
zamnos wrote: post #62

Earlier quoted context omitted.

The error message implies that the compiled default libraries on the M1 don't support the model format, even though it works fine in Paperspace. The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. Traceback (most recent call last): File "/Users/fragmede/projects/llm/dolly/foo.py", line 5, in instruct_pipeline = pipeline( ^^^^^^^^^ File "/Library/Frameworks/Python.fr…

I was referring to his TIL post about setting it up on paperspace, not about apple hardware.

ah, apologies, i misread your comment and was more excited to share since I was able to try on my system.
Post reply on HN