Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running large language models like ChatGPT on a single GPU

github.com

Post reply on HN
Print view View original

121–130 of 274 posts

Re: Running large language models like ChatGPT on a single GPU

#121

Post by anon291 »

gorbypark wrote: post #15

If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.

As a former AI accelerator employee (laid off), I'm kind of happy I was laid off because I realistically don't see a need for specialized hardware anymore.

Large companies can afford Nvidia. Nvidia's software stack is best in class. There's no business need here and the model execution is increasingly becoming possible on single consumer GPUs.

The only place where I see specialized chips excelling is on the edge or if they are truly revolutionary (in which case they're only an acquisition target for Nvidia).

The truth is... The large language models are likely excessively large.

Re: Running large language models like ChatGPT on a single GPU

#122

Post by Miraste »

lxe wrote: post #109
Miraste wrote: post #71

Earlier quoted context omitted.

You have to change the --percent flag. It takes some experimentation. The format is three pairs of 0-100 integers, one for parameters, attention cache and hidden states respectively. The first zero is percent on GPU, the second one is percent on CPU (system RAM), and the remaining percentage will go on disk. For disk offloading to work you may also have to specify --offload-dir. I have opt-30B running on a 3090 with…

How much system RAM are you running with? And I'm guessing it wouldn't hurt to have a fast SSD for disk offloading?

128GB, but by turning on compression I managed to fit the whole thing on the GPU. I did try it off a mix of RAM and SSD as well, and it was slower but still usable. Presumably disk speed matters a lot.

Re: Running large language models like ChatGPT on a single GPU

#123

Post by Miraste »

bioemerl wrote: post #12
benlivengood wrote: post #4

This also means local fine-tuning is possible. Expect to see an explosion of new things like we did with Stable Diffusion, limited to some extent by the ~0.7 order of magnitude more VRAM required.

Does it? I would have expected compression losses to make training really hard.

The compression is optional.

Re: Running large language models like ChatGPT on a single GPU

#124

Post by Miraste »

fathyb wrote: post #75
MuffinFlavored wrote: post #13

Earlier quoted context omitted.

I tried to figure out how to do GPGPU stuff as a total beginner in Rust on Apple Silicon. I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. It also didn't seem to make much of a difference because while there are crates for both that seemed relatively well-maintained/fleshed out, I couldn't figure out how exactly to just pull one down and plug them into a higher level library…

> I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. If you want cross-platform compatibility (kinda), go for OpenCL, if you want the best performance go for Metal. Both use a very similar language for kernels, but Metal is generally more efficient. > Have you had any luck? Not in ML, but I'm doing a lot of GPGPU on Metal, I recently started doing it in Rust. A bit less conveni…

I'm not familiar with Metal, but on Apple Silicon aren't CPU and GPU memory completely shared?

Re: Running large language models like ChatGPT on a single GPU

#125

Post by Miraste »

dharma1 wrote: post #92

I’d love to run this on a single 24gb 3090 - how much dram / SSD space do I need for a decent LLM, when it’s quantised to 4bits?

I've been trying this, and with compression on (4 bits) you can fit the entire 30B model on the 3090.

Re: Running large language models like ChatGPT on a single GPU

#126

Post by fancyfredbot »

ricardobeat wrote: post #116
fancyfredbot wrote: post #97

Earlier quoted context omitted.

I believe that you can't get enough RAM with M1/M2 for this to be useful

This is meant to run on GPUs with 16GB RAM. Most M1/M2 users have at least 32GB (unified memory), and you can configure a MBP or Mac Studio with up to 96/128GB. The Mac Pro is still Intel, but it can be configured with up to 1.5TB of RAM, you can imagine the M* replacement will have equally gigantic options when it comes out.

If you look closely there's 16GB of GPU memory and over 200GB of CPU memory. So none of the currently available M* have the same kind of capacity. Let's hope this changes in the future!

Re: Running large language models like ChatGPT on a single GPU

#127

Post by mensetmanusman »

neilmovva wrote: post #118
gaogao wrote: post #33

Earlier quoted context omitted.

Meta has released the model weights for OPT-175B, which is used in the paper. There's also a lot of full release LLMs from other labs on the way as well.

While OPT-175B is great to have publicly available, it needs a lot more training to achieve good results. Meta trained OPT on 180B tokens, compared to 300B that GPT-3 saw. And the Chinchilla scaling laws suggest that almost 4T tokens would be required to get the most bang for compute buck. And on top of that, there are some questions on the quality of open source data (The Pile) vs OpenAI’s proprietary dataset, which…

It’s fun to think about a few billion weights being the difference between useless and gold.

Re: Running large language models like ChatGPT on a single GPU

#128

Post by Miraste »

nodja wrote: post #107
stevofolife wrote: post #76

Out of curiosity, why aren't we crowd sourcing distributed training of LLMs where anyone can join by bringing their hardware or data? Moreover find a way to incorporate this into a blockchain so there is full transparency but also add in differential privacy to protect every participant. Am I being too crazy here?

https://petals.ml/

Petals doesn't train new models, it only runs BLOOM in a distributed way.

Re: Running large language models like ChatGPT on a single GPU

#129

Post by permo-w »

gorbypark wrote: post #15

If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.

do/did you seriously think that the processing requirement was going to “kill” that possibility?

the history of computing clearly indicates that either the requirements for running or the bar for owning this technology was always going to drop

Re: Running large language models like ChatGPT on a single GPU

#130

Post by fathyb »

MuffinFlavored wrote: post #95
fathyb wrote: post #75

Earlier quoted context omitted.

> I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. If you want cross-platform compatibility (kinda), go for OpenCL, if you want the best performance go for Metal. Both use a very similar language for kernels, but Metal is generally more efficient. > Have you had any luck? Not in ML, but I'm doing a lot of GPGPU on Metal, I recently started doing it in Rust. A bit less conveni…

so write a kernel in OpenCL, then call it from Rust is that what machine learning is doing at a high level?

At a very high level yes. There is also the very important step of efficiently laying out data in the GPU memory to compute tensor values in the kernels.
Post reply on HN