Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Could you train a ChatGPT-beating model for $85k and run it in a browser?

simonwillison.net

Post reply on HN
Print view View original

161–170 of 175 posts

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#161

Post by mirekrusin »

mirekrusin wrote: post #90
amitport wrote: post #86

Earlier quoted context omitted.

hmmm... seems like you're reinventing distributed learning. merge-ability does exist and you can average the results.

You can if you have same base weights. If you have similar variants of the same task you can accelerate it more where the diff is. You can't average on past results computed from historic base weights - it's linear process. If you could do that, you'd just map training examples to diffs and merge them all. Or take two distinct models and merge them to have model that is roughly sum of them. You can't do it, it's not…

I did some bad use of words there "it's linear process" + "it's not linear process" :)

Let me clarify:

It's serialised, iterative, step repeating process where each step depends on output of previous one - aka linear process.

Where each step is non-linear transformation (gradient descent).

It's not distributable (over internet) task because it'd require transferring gigabytes of data (whole model weights) on each step.

To put it in other words - distributed task has massive input size and requires quick computation and tasks arrive very frequently - which means it can't be distributed over internet.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#162

Post by hnav »

SomewhatLikely wrote: post #157
ftxbro wrote: post #14

His estimate is that you could train a LLaMA-7B scale model for around $82,432 and then fine-tune it for a total of less than $85K. But when I saw the fine tuned LLaMA-like models they were worse in my opinion even than GPT-3. They were like GPT-2.5 or like that. Not nearly as good as ChatGPT 3.5 and certainly not ChatGPT-beating. Of course, far enough in the future you could certainly run one in the browser for $85K…

The crazy thing to me is that this means we're approaching being able to have a huge chunk of human knowledge just sitting there locally on your machine. I asked ChatGPT 4 about my old professor and it was able to write a few paragraphs on her including some very specific details. It's like you can fit most of the value of a search engine AND the retrieved pages into a quite small hardware footprint.

it can't be factual though, otherwise you'll have found compression with infinite ratio. I think the next step is a model that can say "idk" rather than coming up with bullshit

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#163

Post by cleanchit »

whalesalad wrote: post #49

Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.

This is how you get skynet.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#165

Post by xsmasher »

robertlagrant wrote: post #35
icelancer wrote: post #25

Earlier quoted context omitted.

Yeah, the constant barrage of "THIS IS AS GOOD AS CHATGPT AND IS PRIVATE" screeds from LLaMA-based marketing projects are getting ridiculous. They're not even remotely close to the same quality. And why would they be? I want the best LLMs to be open source too, but I'm not delusional enough to make insane claims like the hundreds of GitHub forks out there.

> I want the best LLMs to be open source too How do you do this without being incredibly wealthy?

A company that wants to sell you the hardware that LLMs run on might do this. NVIDIA? Apple?

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#166

Post by robertlagrant »

ftxbro wrote: post #51
robertlagrant wrote: post #35

Earlier quoted context omitted.

> I want the best LLMs to be open source too How do you do this without being incredibly wealthy?

Yes a rugged individual would have to be incredibly wealthy to do it! But maybe the governments will make one and maintain it with taxes as an infrastructure service, like roads, giving everyone expanded powers of cognition, memory, and expertise, and raising the consciousnesses of humanity to new heights. Probably in USA it wouldn't happen if we judge ourselves only in zero sum relation to others - helping everyone…

The US spends more on its citizens than almost any other country, and more on helping other countries than any other country.

The problem with making something nationalised or a utility is you'd better have made sure there's no innovation needed or risk required. Once that's all settled, then maybe consider it.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#167

Post by robertlagrant »

simonw wrote: post #46
mejutoco wrote: post #40

Earlier quoted context omitted.

Pooling resources a la SETI@home would be an interesting option I would love to see.

My understanding is that can work for model inference but not for model training. https://github.com/bigscience-workshop/petals is a project that does this kind of thing for running inference - I tried it out in Google Collab and it seemed to work pretty well. Model training is much harder though, because it requires a HUGE amount of high bandwidth data exchange between the machines doing the training - way more than…

And a lot of expensive data scientists.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#168

Post by captainmuon »

I guess companies like OpenAI and Google have no incentives to make models use less resources. The compute required, and of course also their training data, is their moat.

If you accept that your model knows less about the world - it doesn't have to know about every restaurant in mexico city or the biography of every soccer player around the world - then you can get away with much fewer parameters and much less training data. Then you can't query it like an oracle about random things anymore, but you shouldn't do that anyway. But it should still be able to do tasks like reformulating texts, judging simularity (by embedding distance), and so on.

And TFA mentions it also, you could hook up your simple language model with something like ReAct to get really good results. I don't see it running in the browser, but if you had a license-wise clean model that you can run on premises on one or two GPUs, that would be huge for a lot of people!

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#169

Post by Metus »

bitL wrote: post #127
Metus wrote: post #30

Earlier quoted context omitted.

There is a video model making the rounds on /r/stablediffusion and it is just a tiny bit larger than Stable Diffusion.

It's cool though not very stable in details over temporal axis.

Of course the quality is horrible relative to a proper video, it just illustrates that txt2vid might not need 100B+ parameters.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#170

Post by d4rkp4ttern »

Everyone seems to assume that all the “tricks” behind training ChatGPT are known. The only clues are in papers from ClosedAI like the InstructGPT paper. So we assume there is Supervised Fine Tuning, then Reward Modeling and finally RLHF.

But there are most likely other tricks that ClosedAI has not published. These probably took years of R&D to come up with, others trying to replicate ChatGPT would need to come up with these tricks on their own.

Also curiously the app was released in late 2022 while the knowledge cutoff is 2021 — I was curious why that might be, and one hypothesis I had was that it may have been because they wanted to keep the training data fixed while they iterated on numerous methods, hyperparameter tuning etc. All of these are unfortunately a defensive moat that ClosedAI has.

Post reply on HN