Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running large language models like ChatGPT on a single GPU

github.com

Post reply on HN
Print view View original

141–150 of 274 posts

Re: Running large language models like ChatGPT on a single GPU

#141

Post by rnosov »

stevofolife wrote: post #76

Out of curiosity, why aren't we crowd sourcing distributed training of LLMs where anyone can join by bringing their hardware or data? Moreover find a way to incorporate this into a blockchain so there is full transparency but also add in differential privacy to protect every participant. Am I being too crazy here?

The problem here is that most people just don't have suitable hardware. Ideally, you'd want to load the entire model into a GPU and most consumer grade GPUs just don't have nowhere near enough video memory. You'd need to have something like A100 80GB GPU to be able to run a node in the potential blockchain. You can buy one of these cards for about 15k USD. Admittedly, that's not that too far off from the price of a modern bitcoin ASIC miner but still a healthy chunk of change.

And if you try to split the model across several GPUs then you'll have an issue of bandwidth as model parts would need to talk to each other (on the order of terabyte/second). At the moment, the only realistic way to contribute is just to provide feedback data for the RLHF training.

Re: Running large language models like ChatGPT on a single GPU

#142

Post by nodja »

Miraste wrote: post #128
nodja wrote: post #107

Earlier quoted context omitted.

https://petals.ml/

Petals doesn't train new models, it only runs BLOOM in a distributed way.

You can finetune with it. If you want a more generic framework you can use hivemind[1] which is what petals uses, but you'll have to create your own community for whatever model you're trying to train.

https://github.com/learning-at-home/hivemind

Re: Running large language models like ChatGPT on a single GPU

#143

Post by paxys »

Ajedi32 wrote: post #84
paxys wrote: post #72

Earlier quoted context omitted.

Because things happen every day. If ChatGPT wants to compete with Google, staying up to date with recent events is the minimum bar.

You wouldn't need to re-train from scratch for that, just fine-tune on the new data sources. I don't think constant re-training is the optimal strategy for that use-case anyway. Bing does it by letting the LLM search a more traditional web index to find the information it needs.

Okay but someone has to do the fine tuning. The code has to be updated. Parts of the training have to be redone. All of this has costs. It isn't a "do it once and forget about it" task that it is being touted as in this thread.

Re: Running large language models like ChatGPT on a single GPU

#144

Post by inciampati »

postalrat wrote: post #110
anononaut wrote: post #102

Earlier quoted context omitted.

I heard it was $4MM alone in AWS compute time.

So about 40k on your own machines.

It does make it seem like a box of H100s will easily be able to make an interesting open LLM.

Re: Running large language models like ChatGPT on a single GPU

#145

Post by blagie »

A lot of people are looking at this wrong. A $350 3060Ti has 12GB RAM. If there's a way to run models locally, it opens up the door to:

1) Privacy-sensitive applications

2) Tinkering

3) Ignoring filters

4) Prototyping

5) Eventually, a bit of extra training

The upside isn't so much cost / performance, as local control over a cloud-based solution.

Re: Running large language models like ChatGPT on a single GPU

#146

Post by strohwueste »

idiotsecant wrote: post #91
celdon25 wrote: post #64

Earlier quoted context omitted.

Probably one where there isn't an intrinsic conflict of interest with AI risk. Or from a more traditional angle, one where the author's vanity isn't required to be appeased in order for users/customers to be happy. I'm of the opinion that you should do something with game-changing technology because the world needs it, not because you need an ego boost. All technology brings side effects, and there is no greater exam…

People often (usually) do objectively useful things because it's in their selfish interests to do so, ego or otherwise. The surest road to failure is expecting people to act virtuously. Generally systems that assume virtue fail, and systems that assume selfish action and steer that selfish action towards the greater good succeed. In other words, I don't care why people do things, only that they do.

The summary includes a dangerous thought: For example: why does north Korea develop a nuclear bomb is not important, just that they do. But only they why makes it problematic.

Re: Running large language models like ChatGPT on a single GPU

#147

Post by p1esk »

throwawayapples wrote: post #89
breckenedge wrote: post #42

Earlier quoted context omitted.

Thanks for running the cloud numbers on this. I ran some DIY numbers and they indicate less than a week to break even with the cloud, including all hardware and electricity costs. The cloud seems stupid expensive compared to running your own hardware for this kind of task.

The cloud is always stupid expensive compared to running your own hardware for almost any sort of task that isn't highly variable upon one or more axis (cpu, ram, etc), but less than a week to break even is truly shocking.

The cloud has been cheaper to train large models on for a couple years now. Compare buying 8xA100 server vs renting one on Lambda Labs. At least 3 years to break even - if you are using it non-stop 24/7. Longer if not.

Re: Running large language models like ChatGPT on a single GPU

#148

Post by lxe »

lxe wrote: post #139
Miraste wrote: post #122

Earlier quoted context omitted.

128GB, but by turning on compression I managed to fit the whole thing on the GPU. I did try it off a mix of RAM and SSD as well, and it was slower but still usable. Presumably disk speed matters a lot.

Well just got some more sticks. While I wait for RAM to arrive, will try with compress_weight and compress_cache. If you're in any discord or any other space where people are tinkering with this, would love to join!

With compression, was able to get 30b to run on 3090 with '100 0'! Let me see if I can tweak the prompt a bit and make it come alive...

Re: Running large language models like ChatGPT on a single GPU

#149

Post by Aperocky »

blagie wrote: post #145

A lot of people are looking at this wrong. A $350 3060Ti has 12GB RAM. If there's a way to run models locally, it opens up the door to: 1) Privacy-sensitive applications 2) Tinkering 3) Ignoring filters 4) Prototyping 5) Eventually, a bit of extra training The upside isn't so much cost / performance, as local control over a cloud-based solution.

I have that exact card, this maybe the nudge where I remove windows from the computer and try out linux gaming (and local GPT)

Re: Running large language models like ChatGPT on a single GPU

#150

Post by inciampati »

stevenhuang wrote: post #74
SekstiNi wrote: post #68

Earlier quoted context omitted.

> Interesting, though apparently the OPT175B model is 350GB: Only in FP16. In the paper they use int4 quantization to reduce it to a quarter of that. In addition to the model weights, there's also a KV cache that takes up considerable amounts of memory, and they use int4 on that as well. > I wonder what FlexGen is doing.. a naive guess is a mix of SSD and system memory. That's correct, but other approaches have done…

I want to emphasize how fascinating I find that the transform from 16 bit to a 4 bit quantization results in negligible performance loss. That's huge. Is the original FP16 not compressed? The allowance for this more granular quantization seems to suggest the "bottleneck" is in some other aspect of the system, and maybe until that is addressed, a higher fidelity quantization does not improve performance. Or maybe it's…

Very insightful! Now I'm curious what the bottleneck is.
Post reply on HN