Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running large language models like ChatGPT on a single GPU

github.com

Post reply on HN
Print view View original

91–100 of 274 posts

Re: Running large language models like ChatGPT on a single GPU

#91

Post by idiotsecant »

celdon25 wrote: post #64
idiotsecant wrote: post #60

Earlier quoted context omitted.

What motivation would be sufficiently noble?

Probably one where there isn't an intrinsic conflict of interest with AI risk. Or from a more traditional angle, one where the author's vanity isn't required to be appeased in order for users/customers to be happy. I'm of the opinion that you should do something with game-changing technology because the world needs it, not because you need an ego boost. All technology brings side effects, and there is no greater exam…

People often (usually) do objectively useful things because it's in their selfish interests to do so, ego or otherwise. The surest road to failure is expecting people to act virtuously. Generally systems that assume virtue fail, and systems that assume selfish action and steer that selfish action towards the greater good succeed.

In other words, I don't care why people do things, only that they do.

Re: Running large language models like ChatGPT on a single GPU

#93

Post by lxe »

Got the ops-6.7b chatbot running on a windows machine with a 3090 in mere minutes. The only difference was to install the cuda pytorch `pip install torch==1.13.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117` just like in stable diffusion's case.

It performs as expected:

    Human: Tell me a joke
    Machine: I have no sense of humour

    Human: What's 2+5?
    Machine: I cannot answer that.

Re: Running large language models like ChatGPT on a single GPU

#94

Post by idiotsecant »

IncRnd wrote: post #66
idiotsecant wrote: post #60

Earlier quoted context omitted.

What motivation would be sufficiently noble?

Noble? You're anthropomorphising machine learning. On possible motiviation would be to train a model, instead of training a model in order to create publicity around a model being trained.

I think you're misreading, nobody is anthropomorphizing anything other than the very 'anthro' component of the system we're talking about - the people distributing the funding.

Re: Running large language models like ChatGPT on a single GPU

#95

Post by MuffinFlavored »

fathyb wrote: post #75
MuffinFlavored wrote: post #13

Earlier quoted context omitted.

I tried to figure out how to do GPGPU stuff as a total beginner in Rust on Apple Silicon. I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. It also didn't seem to make much of a difference because while there are crates for both that seemed relatively well-maintained/fleshed out, I couldn't figure out how exactly to just pull one down and plug them into a higher level library…

> I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. If you want cross-platform compatibility (kinda), go for OpenCL, if you want the best performance go for Metal. Both use a very similar language for kernels, but Metal is generally more efficient. > Have you had any luck? Not in ML, but I'm doing a lot of GPGPU on Metal, I recently started doing it in Rust. A bit less conveni…

so write a kernel in OpenCL, then call it from Rust

is that what machine learning is doing at a high level?

Re: Running large language models like ChatGPT on a single GPU

#96

Post by idiotsecant »

Dylan16807 wrote: post #26
humanistbot wrote: post #22

Earlier quoted context omitted.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

On the other hand, one techie with a few million dollars... And you could train something like GPT-3 for cheaper than a superbowl commercial. That would get you a lot of publicity.

Is there information out there about how much it cost (in time or human-hours) to do the additional training necessary to make chatGPT? I am genuinely curious what the scale of the effort was.

Re: Running large language models like ChatGPT on a single GPU

#98

Post by stavros »

gaogao wrote: post #33
humanistbot wrote: post #22

Earlier quoted context omitted.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

Meta has released the model weights for OPT-175B, which is used in the paper. There's also a lot of full release LLMs from other labs on the way as well.

Are there any that perform anywhere close to GPT-3?

Re: Running large language models like ChatGPT on a single GPU

#99

Post by Filligree »

SomeHacker44 wrote: post #90
cypress66 wrote: post #29

Earlier quoted context omitted.

This is most likely aimed at people running models locally. And a homelab with 3090s/4090s is one or two orders of magnitude cheaper than GCP, if you use them continuously.

I do not know anyone offhand with a 200+GB RAM home computer. The GPU is not all that is needed; you need to keep the parameters and other stuff in memory too.

Running it off a fast NVMe apparently works. I don't know what the performance is like, though.

Re: Running large language models like ChatGPT on a single GPU

#100

Post by leesec »

humanistbot wrote: post #22
gorbypark wrote: post #15

If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

No it isn't. Stable Diffusion is less than 200 grand to train.
Post reply on HN