Live data from Hacker News

Could you train a ChatGPT-beating model for $85k and run it in a browser?

simonwillison.net

71–80 of 175 posts

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#71

Earlier quoted context omitted.

Yeah, the constant barrage of "THIS IS AS GOOD AS CHATGPT AND IS PRIVATE" screeds from LLaMA-based marketing projects are getting ridiculous. They're not even remotely close to the same quality. And why would they be? I want the best LLMs to be open source too, but I'm not delusional enough to make insane claims like the hundreds of GitHub forks out there.

> I want the best LLMs to be open source too How do you do this without being incredibly wealthy?

You (1) are a company who (2) understands the business domain and has an appropriate business plan.

Sadly the reality of funding today makes it unlikely that these two will both be simultaneously satisfied. The problem is that history will look back on the necessary business plan and deem it a failure even if it generates a company that does a billion dollars plus in annual revenue.

This is actually not unique to large language models but most innovation around computers. The basic problem is that if you build a force-multiplier (spreadsheets, personal computing, large-language models all come to mind) then what will make it succeed is its versatility: people want a hammer that can be used for smashing all manner of things, not just your company's particular brand of matching nails. And most people will only pick up that hammer once per week or once per month, only like 1% of the economy if that will be totally revolutionized, "we use this force-multiplier every day, it is now indispensable, we can't imagine life without it," and it's never predictable what that sector will be -- it's going to be like "oh, who ever dreamed that the killer application for LLMs would be them replacing AutoCAD at mechanical contractors" or some shit.

In those strange eons, to wildly succeed, one must give up on anticipating all usages of the software, one must cease controlling it and set it free. "Well where's the profit in that?" -- it is that this company was one of the first players in the overall market, they got an early chance to stake out as much territory as possible. But the market exploded way larger than they could handle and then everybody looks back on them and says "wow, what a failure, they only captured 1% of that market, they could have been so much more successful." Yeah, they captured 1% of a $100B market, some failure, right?

But what actually happens is that companies see the potential, investors get dollar signs in their eyes, everyone starts to lock down and control these, "you may use large language models but only in the ways that we say, through the interfaces which we provide," and then the only thing that you can use it for is to get generic conversational advice about your hemorrhoids, so after 5-10 years the bubble of excitement fizzles out. Nobody ever dreams to apply it to AutoCAD or whatever, and the world remains unchanged.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#72
post #20

Earlier quoted context omitted.

you're comparing the capital cost of acquiring a GPU machine with the operational cost of renting one in the cloud. Ignoring the operational costs of on-prem hardware is pretty common, but those costs are significant and can greatly change the calculation.

Heh, you work at AWS or Google Cloud perhaps? ;) (Only joking about this as I constantly see employees from AWS/GCloud and other cloud providers claim that cloud is always cheaper than hosting things yourself) Sure, if you're planning to service a large number of users, building your infrastructure in-house might be a bit overkill, as you'll need a infrastructure team to service it as well. If you're just want to buy…

I used to work for Google Cloud (I built a predecessor to Preemptible VMs and also launched Google Cloud Genomics). But even before I worked at Google I was a big fan of AWS (EC2 and S3).

Buying and selling hardware isn't free; it comes with its own cost. I would not want to be in the position of selling a $100K box of computer equipment- ever.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#73
post #60

Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.

That’d be cool but I don’t think most idle consumer GPUs (6-8GB) would have large enough memory for a single iteration (batch size 1) of modern LLMs. But I’d love to see more federated/distributed learning platforms.

6GB can store 3 billion parameters, gpt3.5 has 175 billion parameters.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#74
post #27

Alpaca uses knowledge distillation (it's trained on outputs from OpenAI models). It's something to keep in mind. You're teaching your model to copy an other model's outputs.

> You're teaching your model to copy an other model's outputs.

That's a time honoured tradition in ML, invented by the father of the field himself, Geoffrey Hinton, in 2015.

> Distilling the Knowledge in a Neural Network

https://arxiv.org/abs/1503.02531

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#75

So cool it runs on a browser /sarcasm/ i might not even need a computer. Or internet when we are at it. It either runs locally or it runs on the cloud. Data could come from both locations as well. So it's mostly technically irrelevant if it's displaying in a browser or not. Except when it comes to usability. I don't get it why people love software running in a browser. I often close important tools i have not saved w…

Browser is the true edge compute.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#76
post #72

Earlier quoted context omitted.

Heh, you work at AWS or Google Cloud perhaps? ;) (Only joking about this as I constantly see employees from AWS/GCloud and other cloud providers claim that cloud is always cheaper than hosting things yourself) Sure, if you're planning to service a large number of users, building your infrastructure in-house might be a bit overkill, as you'll need a infrastructure team to service it as well. If you're just want to buy…

I used to work for Google Cloud (I built a predecessor to Preemptible VMs and also launched Google Cloud Genomics). But even before I worked at Google I was a big fan of AWS (EC2 and S3). Buying and selling hardware isn't free; it comes with its own cost. I would not want to be in the position of selling a $100K box of computer equipment- ever.

:)

True, but some things are harder to sell than others. A100's in today's market would be easy to sell. Harder to buy, because the supply is so low unless you're Google or another big name, but if you're trying to sell them, I'm sure you can get rid of them quickly.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#77
post #71

Earlier quoted context omitted.

> I want the best LLMs to be open source too How do you do this without being incredibly wealthy?

You (1) are a company who (2) understands the business domain and has an appropriate business plan. Sadly the reality of funding today makes it unlikely that these two will both be simultaneously satisfied. The problem is that history will look back on the necessary business plan and deem it a failure even if it generates a company that does a billion dollars plus in annual revenue. This is actually not unique to lar…

what stage of capitalism is this

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#78

It seems the quality goes up & cost goes down significantly with Colossal AI's recent push: https://medium.com/@yangyou_berkeley/colossalchat-an-open-so... Their writeup makes it sounds like, net, 2X+ over Alpaca, and that's an early run The browser side is interesting too. Browser JS VMs have a memory cap of 1GB, so that may ultimately be the bottleneck here...

does the 1gb limit extend to wasm?

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#79

Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.

A few people have built frameworks to do this.

There is still a very large open problem in how to federate large numbers of loosely coupled computers to speed up training "interesting" models. I've worked in both domains (protein folding via Folding@Home/protein folding using supercomputers, and ML training on single nodes/ML training on supercomputers) and at least so far, ML hasn't really been a good match for embarrassingly parallel compute. Even in protein folding, folding@home has a number of limitations that are much better addressed on supercomputers (for example: if your problem requires making extremely long individual simulations of large proteins).

All that could change, but I think for the time being, interesting/big models need to be trained on tightly coupled GPUs.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#80
post #79

Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.

A few people have built frameworks to do this. There is still a very large open problem in how to federate large numbers of loosely coupled computers to speed up training "interesting" models. I've worked in both domains (protein folding via Folding@Home/protein folding using supercomputers, and ML training on single nodes/ML training on supercomputers) and at least so far, ML hasn't really been a good match for emba…

Probably going to mirror the transition from single-threaded to multi-threaded compute. Took a while until application architectures took hold of the populous to utilize multi-core.
Post reply on HN