Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running large language models like ChatGPT on a single GPU

github.com

Post reply on HN
Print view View original

41–50 of 274 posts

Re: Running large language models like ChatGPT on a single GPU

#41

Post by adamnemecek »

I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834

The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback.

I argue that autodiff can be replaced with what I call in the paper Hopf coherence which happens within the single layer as opposed to across the whole graph.

Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella.

I'm working on a next gen Hopf algebra based machine learning framework.

Join my discord if you want to discuss this further https://discord.gg/mr9TAhpyBW

Re: Running large language models like ChatGPT on a single GPU

#42

Post by breckenedge »

muttled wrote: post #20

This is cool! But I wonder if it's economical using cloud hardware. The author claims 1.12 tokens/second on the 175B parameter model (arguably comparable to GPT-3 Davinci). That's about 100k tokens a day on the GCP machine the author used. Someone double check my numbers here, but given the Davinci base cost of $0.02 per 1k tokens and GCP cost for the hardware listed "NVIIDA T4 (16GB) instance on GCP with 208GB of DR…

Thanks for running the cloud numbers on this. I ran some DIY numbers and they indicate less than a week to break even with the cloud, including all hardware and electricity costs. The cloud seems stupid expensive compared to running your own hardware for this kind of task.

Re: Running large language models like ChatGPT on a single GPU

#43

Post by polishdude20 »

tarr11 wrote: post #32
humanistbot wrote: post #22

Earlier quoted context omitted.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

Wonder if someone would be willing to start an open source project where we could crowdsource donations for training, and people could possibly donate their GPU usage for it.

There's gotta be something like this already. Like a SETI @ Home type of thing .

Re: Running large language models like ChatGPT on a single GPU

#44

Post by JoshCole »

humanistbot wrote: post #22
gorbypark wrote: post #15

If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

No, Stable Diffusion isn't the only one to release their weights. OpenAI hasn't been releasing weights for ChatGPT, but Stable Diffusion isn't the only ones releasing weights [1].

[1]: https://huggingface.co/

Re: Running large language models like ChatGPT on a single GPU

#45

Post by celdon25 »

Dylan16807 wrote: post #26
humanistbot wrote: post #22

Earlier quoted context omitted.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

On the other hand, one techie with a few million dollars... And you could train something like GPT-3 for cheaper than a superbowl commercial. That would get you a lot of publicity.

I would hope publicity isn’t the motivation for doing it though.

Re: Running large language models like ChatGPT on a single GPU

#46

Post by nirav72 »

cypress66 wrote: post #21
birdyrooster wrote: post #8

I recently bought a T4 to go with my epyc 7402 and 512GB ram for fun and this looks like a great use case. Thanks!

What's the advantage of purchasing a T4 instead of a 3090 or 4090?

Possibly the price. On secondary markets like Ebay - I've occasionally seen T4 cards for $500-600. Also, the form factor. The T4s are comparatively much smaller/shorter than a 3090/4090. So would be a easier fit in a server case.

Re: Running large language models like ChatGPT on a single GPU

#47

Post by scottmf »

tarr11 wrote: post #32
humanistbot wrote: post #22

Earlier quoted context omitted.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

Wonder if someone would be willing to start an open source project where we could crowdsource donations for training, and people could possibly donate their GPU usage for it.

That’s what Stability AI has been doing… There are already open source LLMs the size of GPT-3 such as OPT and Bloom

Re: Running large language models like ChatGPT on a single GPU

#49

Post by dom96 »

It's really interesting that these models are written in Python. Anyone know how much of a speed up using a faster language here would have? Maybe it's already off-loading a lot of the computation to C (I know many Python libraries do this), but I'd love to know.

Re: Running large language models like ChatGPT on a single GPU

#50

Post by TimeBearingDown »

NavinF wrote: post #39
danuker wrote: post #5

Any chance these work on CPUs with any acceptable performance? I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...

> 10-core 20-thread monster CPU With what, 50GB/s memory bandwidth? That's no monster. The two consumer GPUs in my machine both do 1TB/s and are still bottlenecked on memory bandwidth. > only works with the large proprietary driver here In practice, nothing works without the proprietary driver so this isn't specific to temperature. Also the setting you're looking for is almost certainly `nvidia-smi -pl $watts` for se…

That power limit control is explained in detail a few paragraphs down on that wiki page.

https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Cust...

Post reply on HN