Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running large language models like ChatGPT on a single GPU

github.com

Post reply on HN
Print view View original

101–110 of 274 posts

Re: Running large language models like ChatGPT on a single GPU

#101

Post by celdon25 »

idiotsecant wrote: post #91
celdon25 wrote: post #64

Earlier quoted context omitted.

Probably one where there isn't an intrinsic conflict of interest with AI risk. Or from a more traditional angle, one where the author's vanity isn't required to be appeased in order for users/customers to be happy. I'm of the opinion that you should do something with game-changing technology because the world needs it, not because you need an ego boost. All technology brings side effects, and there is no greater exam…

People often (usually) do objectively useful things because it's in their selfish interests to do so, ego or otherwise. The surest road to failure is expecting people to act virtuously. Generally systems that assume virtue fail, and systems that assume selfish action and steer that selfish action towards the greater good succeed. In other words, I don't care why people do things, only that they do.

That’s fine, as long as publicity isn’t the motivation. It’s safe to assume that isn’t optimal for a projects success (Satoshi understood this). Not sure where you got the idea that the inverse of that was beneficial to such a project. I’ve seen first hand where it becomes a problem.

I’m not aware of many examples of starry-eyed divas achieving great results. Usually you hear about them but only because they are exceptional cases, not the norm. It’s a matter of practicality and not virtue (to say otherwise is purely a straw man argument).

Re: Running large language models like ChatGPT on a single GPU

#102

Post by anononaut »

leesec wrote: post #100
humanistbot wrote: post #22

Earlier quoted context omitted.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

No it isn't. Stable Diffusion is less than 200 grand to train.

I heard it was $4MM alone in AWS compute time.

Re: Running large language models like ChatGPT on a single GPU

#103

Post by exebook »

mryab wrote: post #56
polishdude20 wrote: post #43

Earlier quoted context omitted.

There's gotta be something like this already. Like a SETI @ Home type of thing .

There is! See https://petals.ml/ for inference of models like BLOOM-176B over the internet or https://arxiv.org/abs/2301.11913 and https://arxiv.org/abs/2206.01288 that show you how to do pretraining from scratch in the same setting. Disclaimer: I'm a coauthor of these systems (including the one in OP)

Amazing work! If I had a GPU, I'd join. I know similar project for text-to-image: https://aqualxx.github.io/stable-ui/

Re: Running large language models like ChatGPT on a single GPU

#104

Post by huijzer »

leesec wrote: post #100
humanistbot wrote: post #22

Earlier quoted context omitted.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

No it isn't. Stable Diffusion is less than 200 grand to train.

According to Christopher Potts (Stanford Professor and Chair, Department of Linguistics, and Professor, by courtesy, Department of Computer Science), training a large language model costs about 50 million [1].

[1]: https://youtu.be/-lnHHWRCDGk?t=637

Re: Running large language models like ChatGPT on a single GPU

#105

Post by smoldesu »

MuffinFlavored wrote: post #40
smoldesu wrote: post #19

Earlier quoted context omitted.

Give this a look: https://github.com/guillaume-be/rust-bert https://github.com/guillaume-be/rust-bert/blob/master/exampl... If you have Pytorch configured correctly, this should "just work" for a lot of the smaller models. It won't be a 1:1 ChatGPT replacement, but you can build some pretty cool stuff with it. > it's basically Python or bust in this space More or less, but that doesn't have to be a bad thing. If you'…

To clarify, > Port of Hugging Face's Transformers library, using the tch-rs crate and pre-processing from rust-tokenizers. > tch-rs: Rust bindings for the C++ api of PyTorch. Which "backend" does this end up using on Apple Silicon, MPS (Metal Performance Shaders) or OpenCL? https://pytorch.org/docs/stable/notes/mps.html I'm going to guess MPS?

Whatever your Pytorch install is designed to accelerate. I've got Ampere-accelerated Pytorch running it on my ARM server, I assume MPS is used on compatible systems.

Re: Running large language models like ChatGPT on a single GPU

#107

Post by nodja »

stevofolife wrote: post #76

Out of curiosity, why aren't we crowd sourcing distributed training of LLMs where anyone can join by bringing their hardware or data? Moreover find a way to incorporate this into a blockchain so there is full transparency but also add in differential privacy to protect every participant. Am I being too crazy here?

https://petals.ml/

Re: Running large language models like ChatGPT on a single GPU

#108

Post by zeknife »

moffkalast wrote: post #77
gorbypark wrote: post #15

If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.

This may be a flawed approach, but an interesting idea would be to use the current models as a preprocessor to generate a huge "labelled" dataset of inputs and outputs, and then using that more accurate and specific data to train a smaller one that would fit. It likely wouldn't have nearly as much general knowledge since the data just wouldn't be there but the behaviour could be similar?

Look up knowledge distillation

Re: Running large language models like ChatGPT on a single GPU

#109

Post by lxe »

Miraste wrote: post #71
baobabKoodaa wrote: post #48

I just tried to run the example in the README, using the OPT-30B model. It appeared to download 60GiB of model files, and then it attempted to read all of it into RAM. My laptop has "only" 32GiB of RAM so it just ran out of memory.

You have to change the --percent flag. It takes some experimentation. The format is three pairs of 0-100 integers, one for parameters, attention cache and hidden states respectively. The first zero is percent on GPU, the second one is percent on CPU (system RAM), and the remaining percentage will go on disk. For disk offloading to work you may also have to specify --offload-dir. I have opt-30B running on a 3090 with…

How much system RAM are you running with? And I'm guessing it wouldn't hurt to have a fast SSD for disk offloading?
Post reply on HN