Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running large language models like ChatGPT on a single GPU

github.com

Post reply on HN
Print view View original

181–190 of 274 posts

Re: Running large language models like ChatGPT on a single GPU

#181

Post by nl »

huijzer wrote: post #104
leesec wrote: post #100

Earlier quoted context omitted.

No it isn't. Stable Diffusion is less than 200 grand to train.

According to Christopher Potts (Stanford Professor and Chair, Department of Linguistics, and Professor, by courtesy, Department of Computer Science), training a large language model costs about 50 million [1]. [1]: https://youtu.be/-lnHHWRCDGk?t=637

Yeah this is way wrong, unless counting the salaries of everyone involved for a few years in the lead up while writing software that ended up being used

https://bigscience.huggingface.co/blog/bloom was trained with a $7M grant, and that was the first time they'd done it (so there was a lot of failures).

Re: Running large language models like ChatGPT on a single GPU

#182

Post by nl »

anon291 wrote: post #159
foobiekr wrote: post #132

Earlier quoted context omitted.

Power is the main reason to do custom ASICs. I’d be curious as to your opinion of Recogni given they are claiming a 10x power reduction per unit compute.

Unfortunately, I've worked at several players which promise power reductions. It doesn't matter though. People don't care about cost at this point. If you are cost-sensitive you're not doing the kind of revolutionary AI work these companies need to create a competitive moat. And once your model works on NVIDIA and is trained, how much are you going to spend on ML engineers to make it work on something else? Because t…

This is exactly right.

The only exception is running things on mobile. There is demand for porting models to run natively on mobile, and somewhat reasonable support for doing this.

Re: Running large language models like ChatGPT on a single GPU

#183

Post by cardine »

p1esk wrote: post #174
cardine wrote: post #172

Earlier quoted context omitted.

I'd suggest finding a cheaper vendor if that is the lowest price you can get for an 8xA100 server. We spend a lot on both and colo our servers so I've definitely done the math!

Six months ago I've contacted 12 different vendors, the quotes for four 8xA100 servers ranged from 130k to 200k each. You probably wouldn't want to buy from the low end vendors. Keep in mind, there are three important advantages of cloud: 1. You only pay for what you use (hourly). What is utilization of your on-prem servers? 2. You don't have to pay upfront - easier to ask for budget 3. You can upgrade your hardware…

I know how much we paid and it is substantially less than what you were quoted - very likely from one of the 12 providers you contacted.

It is likely you just didn't realize how much margin these providers have and did not negotiate enough. How else do you think cloud providers are able to afford the rates they are giving? The way you describe it, places like Coreweave are operating as a charity. That isn't true - they just got better prices than you.

Our inference setup is 7 figures, has been running for a while (with new servers purchased frequently along the way) and there have been no issues - the cards, CPU, RAM, are all top of the line server hardware.

1. For inference (which is 80%+ of our need) our utilization is 100% 24/7/365. For stuff that is variable (like training) we often do use cloud - as I mentioned we do both.

2. I am the CEO so I am not sure who I'm asking for budget?

3. At this point we would have paid more for cloud than what we spent purchasing our own hardware. There is nothing stopping us from getting new hardware or cloud with newer cards while still getting to own our current hardware. In fact since our costs over the last year were lower due to us buying our own hardware it is actually easier for us to afford newer cards.

Re: Running large language models like ChatGPT on a single GPU

#184

Post by eternalban »

humanistbot wrote: post #22
gorbypark wrote: post #15

If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

A few million dollars. Kickstart the project, get 100 a head and 100,000 backers. Also check with Uncle Sam and see if there are any grants that can be used for this. Start a campaign and get rich concerned people to donate. Jeff may also want to show AWS can also train AI so maybe even get a break there an Amazon can get some nice PR. The list of possibilities seems extensive given the price tag of $12MM and upside of a fully public GPT.

Re: Running large language models like ChatGPT on a single GPU

#185

Post by borzunov »

muttled wrote: post #20

This is cool! But I wonder if it's economical using cloud hardware. The author claims 1.12 tokens/second on the 175B parameter model (arguably comparable to GPT-3 Davinci). That's about 100k tokens a day on the GCP machine the author used. Someone double check my numbers here, but given the Davinci base cost of $0.02 per 1k tokens and GCP cost for the hardware listed "NVIIDA T4 (16GB) instance on GCP with 208GB of DR…

I'm afraid that, unlike proprietary APIs and Petals, this system can't be used for single-batch inference of 175B models with interactive speeds - the thing you actually need for running ChatGPT and other interactive LM apps. See https://news.ycombinator.com/item?id=34874976

Re: Running large language models like ChatGPT on a single GPU

#186

Post by mightytravels »

lxe wrote: post #155

Got the ops-30b chatbot running on 3090 24GB. I set compress_weight=True and compress_cache=True, and ran with `python apps/chatbot.py --model facebook/opt-30b --percent 100 0 100 0 100 0`. I also modified the prompt a bit to make it more... uh alive: Assistant: Did you know that Saturn is 97 times the size of Earth? Human: Are you sure? Assistant: What difference does size make, really, anyway? Human: You didn't ans…

Awesome - how many GB of hard drive/ SSD was required? What is your CPU RAM?

Re: Running large language models like ChatGPT on a single GPU

#187

Post by MuffinFlavored »

fathyb wrote: post #130
MuffinFlavored wrote: post #95

Earlier quoted context omitted.

so write a kernel in OpenCL, then call it from Rust is that what machine learning is doing at a high level?

At a very high level yes. There is also the very important step of efficiently laying out data in the GPU memory to compute tensor values in the kernels.

Can you confirm if OpenCL has been deprecated going forward for Apple Silicon please?

Also, should I be able to expect to use OpenCL version 3.0 on Apple Silicon, or only v1.2 or 2.0 or something else?

Re: Running large language models like ChatGPT on a single GPU

#188

Post by MuffinFlavored »

fathyb wrote: post #75
MuffinFlavored wrote: post #13

Earlier quoted context omitted.

I tried to figure out how to do GPGPU stuff as a total beginner in Rust on Apple Silicon. I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. It also didn't seem to make much of a difference because while there are crates for both that seemed relatively well-maintained/fleshed out, I couldn't figure out how exactly to just pull one down and plug them into a higher level library…

> I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. If you want cross-platform compatibility (kinda), go for OpenCL, if you want the best performance go for Metal. Both use a very similar language for kernels, but Metal is generally more efficient. > Have you had any luck? Not in ML, but I'm doing a lot of GPGPU on Metal, I recently started doing it in Rust. A bit less conveni…

> GPGPU is not doing ML

> General-purpose computing on graphics processing units

> machine learning

Could you expand on why this is the case please? I thought machine learning was basically brute forcing a bunch of possibilities and keeping track of how different inputs "score", then ranking them accordingly to help make educated predictions later.

> GPGPU (General-Purpose Graphics Processing Unit) and machine learning are not the same thing, although they can be related in some ways.

> GPGPU refers to using the parallel processing power of graphics processing units (GPUs) to perform computations beyond graphics rendering. This involves using the massive number of cores in modern GPUs to accelerate tasks such as scientific simulations, numerical analysis, and other data-intensive applications. Essentially, GPGPU involves leveraging the processing power of GPUs for general-purpose computing tasks, not just for graphics processing.

> On the other hand, machine learning involves using algorithms and statistical models to enable computer systems to learn from data and improve their performance on a specific task. It involves feeding large amounts of data to a machine learning algorithm so that it can learn to recognize patterns and make predictions or decisions based on that data.

> While GPGPU can be used to accelerate the computation required in machine learning tasks, they are not the same thing. Machine learning is a specific type of computation, whereas GPGPU is a technique for accelerating computation in general. Additionally, GPGPU can be used for a wide variety of computational tasks, not just machine learning.

Re: Running large language models like ChatGPT on a single GPU

#189

Post by nl »

Miraste wrote: post #128
nodja wrote: post #107

Earlier quoted context omitted.

https://petals.ml/

Petals doesn't train new models, it only runs BLOOM in a distributed way.

Obviously fine-tuning is just a special (easier) case of training.

They are working on complete training of large models too: https://github.com/yandex-research/swarm

Post reply on HN