Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running large language models like ChatGPT on a single GPU

github.com

Post reply on HN
Print view View original

11–20 of 274 posts

Re: Running large language models like ChatGPT on a single GPU

#11

Post by ml_basics »

Very cool.

Worth mentioning though that the highlighted figures (1.12 tok/s for OPT-175B for "FlexGen with Compression") are for inputs of 512 tokens and outputs of 32 tokens.

Since decoder-only transformer memory requirements scale with the square of sequence lengths, things would probably slow down significantly for very long sequences, which would be required for a back-and-forth conversation.

Still though, until reading this i had no idea that running such a model on-device was remotely feasible!

Re: Running large language models like ChatGPT on a single GPU

#12

Post by bioemerl »

benlivengood wrote: post #4

This also means local fine-tuning is possible. Expect to see an explosion of new things like we did with Stable Diffusion, limited to some extent by the ~0.7 order of magnitude more VRAM required.

Does it? I would have expected compression losses to make training really hard.

Re: Running large language models like ChatGPT on a single GPU

#13

Post by MuffinFlavored »

simonw wrote: post #2

Top item on the roadmap: "Support Apple silicon M1/M2 deployment"

I tried to figure out how to do GPGPU stuff as a total beginner in Rust on Apple Silicon.

I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. It also didn't seem to make much of a difference because while there are crates for both that seemed relatively well-maintained/fleshed out, I couldn't figure out how exactly to just pull one down and plug them into a higher level library (or find said higher level library all together).

Have you had any luck? In my experience, it's basically Python or bust in this space despite lots of efforts to make it not that way?

I also got confuses as to whether a 'shader' was more for the visual GPU output of things, or if it was also a building block for model training/networks/machine learning/etc.

Re: Running large language models like ChatGPT on a single GPU

#14

Post by bioemerl »

danuker wrote: post #5

Any chance these work on CPUs with any acceptable performance? I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...

Nope. 20 cores in a CPU, 2000 in a GPU, with much much faster memory and an architecture designed to chew through data as fast as possible.

Re: Running large language models like ChatGPT on a single GPU

#16

Post by adeon »

danuker wrote: post #5

Any chance these work on CPUs with any acceptable performance? I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...

I don't know about these large models but I saw on a random HN comment earlier in a different topic where someone showed a GPT-J model on CPU only: https://github.com/ggerganov/ggml

I tested it on my Linux and Macbook M1 Air and it generates tokens at a reasonable speed using CPU only. I noticed it doesn't quite use all my available CPU cores so it may be leaving some performance on the table, not sure though.

The GPT-J 6B is nowhere near as large as the OPT-175B in the post. But I got the sense that CPU-only inference may not be totally hopeless even for large models if only we got some high quality software to do it.

Re: Running large language models like ChatGPT on a single GPU

#17

Post by metadat »

bioemerl wrote: post #10
metadat wrote: post #7

> Hardware: an NVIIDA T4 (16GB) instance on GCP with 208GB of DRAM and 1.5TB of SSD. Is FlexGen able to take advantage of multiple hundreds of GB of system memory? Or is do these compute instances just come bundled with it and it's a [largely] irrelevant detail?

The OPT175b model is massive. A lot of that system ram probably holds model data.

Interesting, though apparently the OPT175B model is 350GB:

> You will need at least 350GB GPU memory on your entire cluster to serve the OPT-175B model. For example, you can use 4 x AWS p3.16xlarge instances, which provide 4 (instance) x 8 (GPU/instance) x 16 (GB/GPU) = 512GB memory.

https://alpa.ai/tutorials/opt_serving.html

(Scroll down to the second "Note", not far from the top)

I wonder what FlexGen is doing.. a naive guess is a mix of SSD and system memory. Definitely curious about what FlexGen's underlying strategy translates to in terms of actual data paths.

Re: Running large language models like ChatGPT on a single GPU

#18

Post by narrator »

gorbypark wrote: post #15

If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.

No doubt the corporate large language models will use it to make language models that are 10x bigger. However, at least the public will have access to 175B parameter language models which are much more sophisticated than the 6B or so parameter models consumer video cards can currently run.

Re: Running large language models like ChatGPT on a single GPU

#19

Post by smoldesu »

MuffinFlavored wrote: post #13
simonw wrote: post #2

Top item on the roadmap: "Support Apple silicon M1/M2 deployment"

I tried to figure out how to do GPGPU stuff as a total beginner in Rust on Apple Silicon. I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. It also didn't seem to make much of a difference because while there are crates for both that seemed relatively well-maintained/fleshed out, I couldn't figure out how exactly to just pull one down and plug them into a higher level library…

Give this a look:

https://github.com/guillaume-be/rust-bert

https://github.com/guillaume-be/rust-bert/blob/master/exampl...

If you have Pytorch configured correctly, this should "just work" for a lot of the smaller models. It won't be a 1:1 ChatGPT replacement, but you can build some pretty cool stuff with it.

> it's basically Python or bust in this space

More or less, but that doesn't have to be a bad thing. If you're on Apple Silicon, you have plenty of performance headroom to deploy Python code for this. I've gotten this library to work on systems with as little as 2gb of memory, so outside of ultra-low-end use cases, you should be fine.

Re: Running large language models like ChatGPT on a single GPU

#20

Post by muttled »

This is cool! But I wonder if it's economical using cloud hardware. The author claims 1.12 tokens/second on the 175B parameter model (arguably comparable to GPT-3 Davinci). That's about 100k tokens a day on the GCP machine the author used. Someone double check my numbers here, but given the Davinci base cost of $0.02 per 1k tokens and GCP cost for the hardware listed "NVIIDA T4 (16GB) instance on GCP with 208GB of DRAM and 1.5TB of SSD" coming up to about $434 on spot instance pricing, you could simply use the OpenAI API and generate about 723k tokens a day for the same price as running the spot instance (which could go offline at any point due to it being a spot instance).

Running the fine-tuned versions of OpenAI models are approximately 6x more expensive per token. If you were running a fine-tuned model on local commodity hardware, the economies would start to tilt in favor of doing something like this if the load was predictable and relatively constant.

Post reply on HN