Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

github.com

Post reply on HN
Print view View original

81–90 of 90 posts

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#81

Post by simonw »

Garcia98 wrote: post #49
simonw wrote: post #21

I got this model working on a GPU instance, notes here: https://til.simonwillison.net/llms/dolly-2 Anyone managed to run it on an M1/M2 Mac yet?

What's the most cost-effective alternative to Paperspace? I had a nightmarish experience with them last week after my account got locked up twice when I was training a model with a 1.5 GB dataset that somewhere contained the string "Minecraft Server".

I picked them almost at random from the list suggested by this Fast.AI course: https://course.fast.ai/Lessons/lesson9.html#links-from-the-l...

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#82

Post by Daegalus »

zamnos wrote: post #80
Daegalus wrote: post #78

Earlier quoted context omitted.

I was referring to his TIL post about setting it up on paperspace, not about apple hardware.

ah, apologies, i misread your comment and was more excited to share since I was able to try on my system.

No worries, it happens. I will admit the way I answered wasn't clear that I was referring to the linked page and not the question in the post. All good.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#83

Post by samstave »

ankitmathur wrote: post #75
oidar wrote: post #12

Earlier quoted context omitted.

How hard would it be to get dolly running on llama.cpp?

Hey there! I worked on Dolly, and I work on Model Serving at Databricks. DollyV1 is GPT-J-based, so it'll run easily on llama.cpp. DollyV2 is Pythia-based, which is built with the GPT-NeoX library GPT-NeoX is not that different than GPT-J (it also has the rotary embeddings, which llama.cpp supports for GPT-J). I would imagine it's not too heavy of a lift to add NeoX architecture support

Because the firehost of AI/GPT is a lot to try to take in, please ELI5 unpack and provide more definitions for this comment.

-

Thank you.

Just so I am clear, "parameters" refers to the number of total node-relation-connections btwn a single node and its neighbors for that Prompt/Label? Or how would you explain this ELI5 style?

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#84

Post by ankitmathur »

samstave wrote: post #83
ankitmathur wrote: post #75

Earlier quoted context omitted.

Hey there! I worked on Dolly, and I work on Model Serving at Databricks. DollyV1 is GPT-J-based, so it'll run easily on llama.cpp. DollyV2 is Pythia-based, which is built with the GPT-NeoX library GPT-NeoX is not that different than GPT-J (it also has the rotary embeddings, which llama.cpp supports for GPT-J). I would imagine it's not too heavy of a lift to add NeoX architecture support

Because the firehost of AI/GPT is a lot to try to take in, please ELI5 unpack and provide more definitions for this comment. - Thank you. Just so I am clear, "parameters" refers to the number of total node-relation-connections btwn a single node and its neighbors for that Prompt/Label? Or how would you explain this ELI5 style?

Sure! I'll try to briefly summarize though almost certainly will oversimplify. There are a couple of open source language models trained by Eleuther AI - the first one was called GPT-J, and it used some newer model architecture concepts. Subsequently, they released a model architected in the likeness of GPT-3, called GPT-NeoX-20B. Functionally, it was quite similar architecturally to GPT-J, but just with more parameters. Pythia is a model with the same architecture and the same dataset but with different parameter sizes to test scaling laws.

DollyV2 is a Pythia model fine tuned on the Databricks 15K dataset

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#85

Post by sandkoan »

mrtranscendence wrote: post #47
kumarski wrote: post #28

Amazing. Love databricks.

Databricks is fine. I wasn't happy using it until they implemented the ability to work in a git repo, with proper file support, but that's gone some way to making it more usable to me. The interface sucks pretty hard, slowing down and using a significant amount of memory with only modestly high number of cells (where a Jupyterlab notebook would remain very snappy). I also wish there were a better story for local deve…

When you say you wish they had a "better story for local development," what do you mean? What do you wish for?

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#87

Post by ralfn »

cma wrote: post #76
anentropic wrote: post #68

Earlier quoted context omitted.

the GPT-J-6B one is Dolly 1.0, previously released Dolly 2.0 is Pythia-12B fine-tuned on this new dataset on their hugging face page [1] they admit the performance may not be much or any better than the original model (I am guessing this may be a weakness of Pythia-12B, which was intended for model-training research rather than best results) the main point of Dolly 2.0 is the new dataset is unencumbered legally [2] w…

I think there's probably nothing wrong with training on others' ChatGPT transcripts posted on the open web. OpenAI trains on source-available projects with non-commercial terms, so their lawyers have already been over a similar case and decided it should be fine.

Not just that: Imagine OpenAI going to court and establishing the legal precedent that makes their own product illegal.

So OpenAI can claim whatever they like, there is no way they will ever pursue legal actions, unless their intent is to (intentionally) lose the court case to establish the precedent that it is okay to train on random data you scraped from the internet.

We would also get into a weird situation anyhow where it is hard/impossible to prove whether all/some/none of the information in a dataset is curated by humans. So in the worst case, we will have companies work with human curators (but secretly supplement with gray sourced materials) during their training. Just like how its hard to get 100% slave free coffee beans or cacao.

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#88

Post by anentropic »

ralfn wrote: post #87
cma wrote: post #76

Earlier quoted context omitted.

I think there's probably nothing wrong with training on others' ChatGPT transcripts posted on the open web. OpenAI trains on source-available projects with non-commercial terms, so their lawyers have already been over a similar case and decided it should be fine.

Not just that: Imagine OpenAI going to court and establishing the legal precedent that makes their own product illegal. So OpenAI can claim whatever they like, there is no way they will ever pursue legal actions, unless their intent is to (intentionally) lose the court case to establish the precedent that it is okay to train on random data you scraped from the internet. We would also get into a weird situation anyhow…

I don't think it's about things being illegal per se

But that they can sue you because, by making a competing product with data obtained by using their product, you contravened their terms & conditions for using their product

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#89

Post by ankitmathur »

ankitmathur wrote: post #84
samstave wrote: post #83

Earlier quoted context omitted.

Because the firehost of AI/GPT is a lot to try to take in, please ELI5 unpack and provide more definitions for this comment. - Thank you. Just so I am clear, "parameters" refers to the number of total node-relation-connections btwn a single node and its neighbors for that Prompt/Label? Or how would you explain this ELI5 style?

Sure! I'll try to briefly summarize though almost certainly will oversimplify. There are a couple of open source language models trained by Eleuther AI - the first one was called GPT-J, and it used some newer model architecture concepts. Subsequently, they released a model architected in the likeness of GPT-3, called GPT-NeoX-20B. Functionally, it was quite similar architecturally to GPT-J, but just with more paramet…

Augmenting the answer to address your followup: parameters are any trainable variable in a model's definition. Model training is a process where you basically tweak the parameters in your model and then re-evaluate the model on a metric judging its quality. A lot of models consist of matrix multiplication, so if you are multiplying matrix A of size 2x2 with matrix B of size 2x2 and both matrices can we tweaked, then you've got 8 parameters, since you've got 8 numbers that can be tweaked

Re: Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

#90

Post by ralfn »

anentropic wrote: post #88
ralfn wrote: post #87

Earlier quoted context omitted.

Not just that: Imagine OpenAI going to court and establishing the legal precedent that makes their own product illegal. So OpenAI can claim whatever they like, there is no way they will ever pursue legal actions, unless their intent is to (intentionally) lose the court case to establish the precedent that it is okay to train on random data you scraped from the internet. We would also get into a weird situation anyhow…

I don't think it's about things being illegal per se But that they can sue you because, by making a competing product with data obtained by using their product, you contravened their terms & conditions for using their product

But so did they when they scraped the web for content.

That's not within anyone's terms and conditions except Wikipedia.

That's what I mean with precedent. If OpenAI would win that they would be sued in term by Bloomberg for example.

Post reply on HN