Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Could you train a ChatGPT-beating model for $85k and run it in a browser?

simonwillison.net

Post reply on HN
Print view View original

81–90 of 175 posts

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#81

Post by newhouseb »

avereveard wrote: post #47
simonw wrote: post #34

Earlier quoted context omitted.

Yeah, you're right. I wrote this a couple of weeks ago at the height of LLaMA hype, but with further experience I don't think the GPT-3 comparisons hold weight. My biggest problem: I haven't managed to get a great summarization out of a LLaMA derivative that runs on my laptop yet. Maybe I haven't tried the right model or the right prompt yet though, but that feels essential to me for a bunch of different applications…

my biggest problem with these models is that they cannot reliably produce structured data. even davinci can be used as part of a chain, because you can direct it to structure and unstructure data, and then extract the single component and build them into tasks. cohere, llama et al are currently struggling to consistently produce these result reliably, even if you can chat with them and frankly it's not about the chat…

Last weekend I built some tooling that you can integrate with huggingface transformers to force a given model to _only_ output content that validates against a JSON schema [1].

The challenge is that for it to work cost effectively you need to be able to append what is basically a final network layer to the model that is algorithmically designed and until OpenAI exposes the full logits and/or some way to modify them on the fly you're going to be stuck with open source models. I've run things against GPT-2 mostly but it's only list to try LLaMA.

[1] "Structural Alignment: Modifying Transformers (like GPT) to Follow a JSON Schema" @ https://github.com/newhouseb/clownfish

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#83

Post by jcims »

dekhn wrote: post #20
version_five wrote: post #16

If you have ~100k to spend, aren't there options to buy a gpu rather than just blow it all on cloud? How much is an 8xA100 machine? 4xA100 is 75k, 8 is 140k https://shop.lambdalabs.com/deep-learning/servers/hyperplane...

you're comparing the capital cost of acquiring a GPU machine with the operational cost of renting one in the cloud. Ignoring the operational costs of on-prem hardware is pretty common, but those costs are significant and can greatly change the calculation.

No kidding. I worked for a company that had multiple billions of dollars invested in a data center refresh in North America and Europe.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#84

Post by Tepix »

simonw wrote: post #34
ftxbro wrote: post #14

His estimate is that you could train a LLaMA-7B scale model for around $82,432 and then fine-tune it for a total of less than $85K. But when I saw the fine tuned LLaMA-like models they were worse in my opinion even than GPT-3. They were like GPT-2.5 or like that. Not nearly as good as ChatGPT 3.5 and certainly not ChatGPT-beating. Of course, far enough in the future you could certainly run one in the browser for $85K…

Yeah, you're right. I wrote this a couple of weeks ago at the height of LLaMA hype, but with further experience I don't think the GPT-3 comparisons hold weight. My biggest problem: I haven't managed to get a great summarization out of a LLaMA derivative that runs on my laptop yet. Maybe I haven't tried the right model or the right prompt yet though, but that feels essential to me for a bunch of different applications…

Have you tried bigger models? Llama-65B can indeed compete with GPT-3 according to various benchmarks. The next thing would be to get the fine-tuning as good as OpenAI's.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#85

Post by alecco »

Interesting blog but the extrapolations are way overblown. I tried one of the 30bn models and it's not even remotely close to GPT-3.

Don't get me wrong, this is very interesting and I hope more is done in the open models. But let's not over-hype by 10x.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#86

Post by amitport »

mirekrusin wrote: post #63
whalesalad wrote: post #49

Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.

Unfortunately training is not emberassingly parallelisable [0] problem. It would require new architecture. Current models diverge too fast. By the time you'd download and/or calculate your contribution the model would descend somewhere else and your delta would not be applicable - based off wrong initial state. It would be great if merge-ability would exist. It would also likely apply to efficient/optimal shrinking f…

hmmm... seems like you're reinventing distributed learning.

merge-ability does exist and you can average the results.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#87

Post by buzzier »

fswd wrote: post #59

There is somebody finetunin 160m rwkv4 on alpaca on the rwkv discord, I am out of the office and can't link but the person posted in prompt showcase channel

RWKV-v4 Web Demo (169m/430m params) https://josephrocca.github.io/rwkv-v4-web/demo/

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#88

Post by PaulDavisThe1st »

whalesalad wrote: post #80
dekhn wrote: post #79

Earlier quoted context omitted.

A few people have built frameworks to do this. There is still a very large open problem in how to federate large numbers of loosely coupled computers to speed up training "interesting" models. I've worked in both domains (protein folding via Folding@Home/protein folding using supercomputers, and ML training on single nodes/ML training on supercomputers) and at least so far, ML hasn't really been a good match for emba…

Probably going to mirror the transition from single-threaded to multi-threaded compute. Took a while until application architectures took hold of the populous to utilize multi-core.

Probably not. Multicore has been a thing for 30 years (We had a 32 core Sequent Systems and a 64 core KSR-1 at UW CS&E in the early 1990s). Everything about these models has been developed in a multicore computing context, and thus far, it still isn't massively-parallel-distributable. An algorithm can be massively parallel without being sensibly distributable. Change the latency between compute nodes is not always a neutral or even just linear decrease in performance.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#89

Post by isoprophlex »

Metus wrote: post #30
bitL wrote: post #9

Earlier quoted context omitted.

Only for images. People want to generate videos next and those models will be likely GPT-sized.

There is a video model making the rounds on /r/stablediffusion and it is just a tiny bit larger than Stable Diffusion.

You're not kidding! it's far from perfect, but pretty funny still...

https://www.reddit.com/r/StableDiffusion/comments/126xsxu/ni...

Too bad SD learned the Shutterstock watermark so well, lol

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#90

Post by mirekrusin »

amitport wrote: post #86
mirekrusin wrote: post #63

Earlier quoted context omitted.

Unfortunately training is not emberassingly parallelisable [0] problem. It would require new architecture. Current models diverge too fast. By the time you'd download and/or calculate your contribution the model would descend somewhere else and your delta would not be applicable - based off wrong initial state. It would be great if merge-ability would exist. It would also likely apply to efficient/optimal shrinking f…

hmmm... seems like you're reinventing distributed learning. merge-ability does exist and you can average the results.

You can if you have same base weights.

If you have similar variants of the same task you can accelerate it more where the diff is.

You can't average on past results computed from historic base weights - it's linear process.

If you could do that, you'd just map training examples to diffs and merge them all.

Or take two distinct models and merge them to have model that is roughly sum of them. You can't do it, it's not linear process.

Post reply on HN