Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Could you train a ChatGPT-beating model for $85k and run it in a browser?

simonwillison.net

Post reply on HN
Print view View original

151–160 of 175 posts

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#151

Post by icelancer »

mewpmewp2 wrote: post #104
Tepix wrote: post #84

Earlier quoted context omitted.

Have you tried bigger models? Llama-65B can indeed compete with GPT-3 according to various benchmarks. The next thing would be to get the fine-tuning as good as OpenAI's.

I wonder how accurate those benchmarks are in terms of actual problem solving capability. I think there's a major line at which point LLM becomes actually useful and it actually feels like you are speaking to something intelligent and that can be useful for you in terms of productivity etc.

They aren't at all. They are synthetic benchmarks that carry little resemblance to real world experience.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#152

Post by jillesvangurp »

dekhn wrote: post #20
version_five wrote: post #16

If you have ~100k to spend, aren't there options to buy a gpu rather than just blow it all on cloud? How much is an 8xA100 machine? 4xA100 is 75k, 8 is 140k https://shop.lambdalabs.com/deep-learning/servers/hyperplane...

you're comparing the capital cost of acquiring a GPU machine with the operational cost of renting one in the cloud. Ignoring the operational costs of on-prem hardware is pretty common, but those costs are significant and can greatly change the calculation.

The issue with on premise is under utilization and the fact that you need more than just the hardware. You end up buying more hardware than you need and inevitably a portion of it will just sit there idling and depreciating in value. And you don't just need hardware but also investments in your building. GPUs generate a lot of heat. So, you need to get rid of that heat and make sure you beef up your power infrastructure to be able to handle the load. It's not just the GPUs that you pay for. And the equipment is expensive. So you need to invest in security as well.

Cloud pricing is pretty steep and obviously has a fat profit margin but building your own data centers isn't cheap either. Doing this at scale is not something most companies would be very good at either. Which means it probably is quite a bit more expensive relative to what the big cloud providers are doing.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#153

Post by semitones »

whalesalad wrote: post #49

Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.

The main reason an arbitrarily distributed set of compute nodes cannot give you good performance for training a model (even if you have an immodest number of nodes), is that the latency of the inter-node communication will be a massive bottleneck. GPU cloud providers shell out big bucks for ultra fast intra-DC networking via infiniband and the like, and the networking is paid attention to as much (if not more sometimes) than the capabilities of the nodes themselves.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#154

Post by GaggiX »

danielbln wrote: post #108
lxe wrote: post #5

Keep in mind that image transformer models like stable diffusion are generally smaller than language models, so they are easier to fit in wasm space. Also. you can finetune llama-7b on a 3090 for about $3 using LoRA.

Generative image models don't use transformers, they're diffusion models. LLMs are transformers.

Diffusion models can use a transformer architecture, example: DiT. Stable Diffusion is using a U-Net architecture with transformer blocks.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#155

Post by canadianfella »

dekhn wrote: post #79
whalesalad wrote: post #49

Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.

A few people have built frameworks to do this. There is still a very large open problem in how to federate large numbers of loosely coupled computers to speed up training "interesting" models. I've worked in both domains (protein folding via Folding@Home/protein folding using supercomputers, and ML training on single nodes/ML training on supercomputers) and at least so far, ML hasn't really been a good match for emba…

[dead]

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#156

Post by UncleEntity »

cj wrote: post #150
gessha wrote: post #148

Earlier quoted context omitted.

And when everyone buys in, you go private everything and reap the benefits. Brilliant!

Except, don’t go private this time. Appoint an external board not controlled by the CEO and write it into your bylaws.

Where’s the money in that?

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#157

Post by SomewhatLikely »

ftxbro wrote: post #14

His estimate is that you could train a LLaMA-7B scale model for around $82,432 and then fine-tune it for a total of less than $85K. But when I saw the fine tuned LLaMA-like models they were worse in my opinion even than GPT-3. They were like GPT-2.5 or like that. Not nearly as good as ChatGPT 3.5 and certainly not ChatGPT-beating. Of course, far enough in the future you could certainly run one in the browser for $85K…

The crazy thing to me is that this means we're approaching being able to have a huge chunk of human knowledge just sitting there locally on your machine. I asked ChatGPT 4 about my old professor and it was able to write a few paragraphs on her including some very specific details. It's like you can fit most of the value of a search engine AND the retrieved pages into a quite small hardware footprint.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#158

Post by cj »

UncleEntity wrote: post #156
cj wrote: post #150

Earlier quoted context omitted.

Except, don’t go private this time. Appoint an external board not controlled by the CEO and write it into your bylaws.

Where’s the money in that?

Nowhere obvious.

I think there are plenty of rich people who would benefit indirectly from running an OpenAI-like non-profit.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#159

Post by gessha »

cj wrote: post #150
gessha wrote: post #148

Earlier quoted context omitted.

And when everyone buys in, you go private everything and reap the benefits. Brilliant!

Except, don’t go private this time. Appoint an external board not controlled by the CEO and write it into your bylaws.

I’m a pessimist about corporate doing the right thing unless it’s economically aligned to do so.

“This time” happens with open source where there’s typically no economical incentive and people are doing it for the heck of it.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#160

Post by dangond »

captaincrowbar wrote: post #91

The big problem with AI R&D is that nobody can keep up with the big bux companies. It makes this kind of project a bit pointless. Even if you can run a GPT3-equivalent on a web browser, how many people are going to bother (except as a stunt) when GPT4 is available?

Cost is a big reason. It doesn't matter how good the top-of-the-line models are if the cheaper ones suit your needs. Commoditization is great that way. I'd absolutely use an open source GPT-4 in my browser over a pricy closed GPT-5 once we get to that point.
Post reply on HN