Running large language models like ChatGPT on a single GPU

This also means local fine-tuning is possible. Expect to see an explosion of new things like we did with Stable Diffusion, limited to some extent by the ~0.7 order of magnitude more VRAM required.

Re: Running large language models like ChatGPT on a single GPU

#5

Post by danuker » Mon, Feb 20, 2023, 5:54 PM UTC

Any chance these work on CPUs with any acceptable performance?

I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here:

https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...

Re: Running large language models like ChatGPT on a single GPU

#6

Post by winddude » Mon, Feb 20, 2023, 5:55 PM UTC

looks interesting. FYI, the link to your discord in the readme is broken

Re: Running large language models like ChatGPT on a single GPU

#7

Post by metadat » Mon, Feb 20, 2023, 5:55 PM UTC

> Hardware: an NVIIDA T4 (16GB) instance on GCP with 208GB of DRAM and 1.5TB of SSD.

Is FlexGen able to take advantage of multiple hundreds of GB of system memory? Or is do these compute instances just come bundled with it and it's a [largely] irrelevant detail?

Re: Running large language models like ChatGPT on a single GPU

#8

Post by birdyrooster » Mon, Feb 20, 2023, 5:57 PM UTC

I recently bought a T4 to go with my epyc 7402 and 512GB ram for fun and this looks like a great use case. Thanks!

Re: Running large language models like ChatGPT on a single GPU

#9

Post by metadat » Mon, Feb 20, 2023, 5:57 PM UTC

danuker wrote: post #5

Any chance these work on CPUs with any acceptable performance? I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...

Unlikely, because this is an efficient GPU work offloader, not a complete replacement for GPU computation.

Re: Running large language models like ChatGPT on a single GPU

#10

Post by bioemerl » Mon, Feb 20, 2023, 5:57 PM UTC

metadat wrote: post #7

> Hardware: an NVIIDA T4 (16GB) instance on GCP with 208GB of DRAM and 1.5TB of SSD. Is FlexGen able to take advantage of multiple hundreds of GB of system memory? Or is do these compute instances just come bundled with it and it's a [largely] irrelevant detail?

The OPT175b model is massive. A lot of that system ram probably holds model data.