Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running large language models like ChatGPT on a single GPU

github.com

Post reply on HN
Print view View original

71–80 of 274 posts

Re: Running large language models like ChatGPT on a single GPU

#71

Post by Miraste »

baobabKoodaa wrote: post #48

I just tried to run the example in the README, using the OPT-30B model. It appeared to download 60GiB of model files, and then it attempted to read all of it into RAM. My laptop has "only" 32GiB of RAM so it just ran out of memory.

You have to change the --percent flag. It takes some experimentation. The format is three pairs of 0-100 integers, one for parameters, attention cache and hidden states respectively. The first zero is percent on GPU, the second one is percent on CPU (system RAM), and the remaining percentage will go on disk.

For disk offloading to work you may also have to specify --offload-dir.

I have opt-30B running on a 3090 with --percent 20 50 100 0 100 0, although I think those could be tweaked to be faster.

Re: Running large language models like ChatGPT on a single GPU

#72

Post by paxys »

Ajedi32 wrote: post #70
paxys wrote: post #55

Earlier quoted context omitted.

You can do it once, but probably not every day.

Why would you want to retrain it from scratch every day? Stable Diffusion doesn't do that either.

Because things happen every day. If ChatGPT wants to compete with Google, staying up to date with recent events is the minimum bar.

Re: Running large language models like ChatGPT on a single GPU

#73

Post by permo-w »

JoshCole wrote: post #44
humanistbot wrote: post #22

Earlier quoted context omitted.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

No, Stable Diffusion isn't the only one to release their weights. OpenAI hasn't been releasing weights for ChatGPT, but Stable Diffusion isn't the only ones releasing weights [1]. [1]: https://huggingface.co/

yeah there an absolute pile of LLMs that are fully open-source. OpenAI’s GPT2 for one, but also Bloom, OPT, GPT-J, and I’m sure myriad others too

Re: Running large language models like ChatGPT on a single GPU

#74

Post by stevenhuang »

SekstiNi wrote: post #68
metadat wrote: post #17

Earlier quoted context omitted.

Interesting, though apparently the OPT175B model is 350GB: > You will need at least 350GB GPU memory on your entire cluster to serve the OPT-175B model. For example, you can use 4 x AWS p3.16xlarge instances, which provide 4 (instance) x 8 (GPU/instance) x 16 (GB/GPU) = 512GB memory. https://alpa.ai/tutorials/opt_serving.html (Scroll down to the second "Note", not far from the top) I wonder what FlexGen is doing.. a…

> Interesting, though apparently the OPT175B model is 350GB: Only in FP16. In the paper they use int4 quantization to reduce it to a quarter of that. In addition to the model weights, there's also a KV cache that takes up considerable amounts of memory, and they use int4 on that as well. > I wonder what FlexGen is doing.. a naive guess is a mix of SSD and system memory. That's correct, but other approaches have done…

I want to emphasize how fascinating I find that the transform from 16 bit to a 4 bit quantization results in negligible performance loss. That's huge. Is the original FP16 not compressed?

The allowance for this more granular quantization seems to suggest the "bottleneck" is in some other aspect of the system, and maybe until that is addressed, a higher fidelity quantization does not improve performance.

Or maybe it's the relative values/ratio between weights that is important, and as long as the intended ratio between weights can be expressed, the exact precision of the weights themselves may not be important?

Found an interesting paper on this below. There's doubtless heavy research underway in this area

- https://www.researchgate.net/publication/367557918_Understan...

Re: Running large language models like ChatGPT on a single GPU

#75

Post by fathyb »

MuffinFlavored wrote: post #13
simonw wrote: post #2

Top item on the roadmap: "Support Apple silicon M1/M2 deployment"

I tried to figure out how to do GPGPU stuff as a total beginner in Rust on Apple Silicon. I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. It also didn't seem to make much of a difference because while there are crates for both that seemed relatively well-maintained/fleshed out, I couldn't figure out how exactly to just pull one down and plug them into a higher level library…

> I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends.

If you want cross-platform compatibility (kinda), go for OpenCL, if you want the best performance go for Metal. Both use a very similar language for kernels, but Metal is generally more efficient.

> Have you had any luck?

Not in ML, but I'm doing a lot of GPGPU on Metal, I recently started doing it in Rust. A bit less convenient than with Swift/Objective-C, but still possible. Worst case you'll have to add an .mm file and bridge it with `extern "C"`. That said, doing GPGPU is not doing ML, and most ML libraries are in Python.

> I also got confuses as to whether a 'shader' was more for the visual GPU output of things, or if it was also a building block for model training/networks/machine learning/etc.

A shader is basically a function that runs for every element of the output buffer. We generally call them kernels for GPGPU, and shaders (geometry, vertex, fragment) for graphics stuff. You have to write them in a language that kinda looks like C (OpenGL GLSL, DirectX HSL, Metal MSL), but is optimized for the SMT properties of GPUs.

Learning shaders will let you run code on the GPU, to do ML you also need to learn what are tensors, how to compute them on the GPU, and how to build ML systems using them.

I recommend ShaderToy [0] if you want a cool way to understand and play with shaders.

[0]: https://www.shadertoy.com/

Re: Running large language models like ChatGPT on a single GPU

#76

Post by stevofolife »

Out of curiosity, why aren't we crowd sourcing distributed training of LLMs where anyone can join by bringing their hardware or data? Moreover find a way to incorporate this into a blockchain so there is full transparency but also add in differential privacy to protect every participant.

Am I being too crazy here?

Re: Running large language models like ChatGPT on a single GPU

#77

Post by moffkalast »

gorbypark wrote: post #15

If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.

This may be a flawed approach, but an interesting idea would be to use the current models as a preprocessor to generate a huge "labelled" dataset of inputs and outputs, and then using that more accurate and specific data to train a smaller one that would fit.

It likely wouldn't have nearly as much general knowledge since the data just wouldn't be there but the behaviour could be similar?

Re: Running large language models like ChatGPT on a single GPU

#78

Post by ImprobableTruth »

yorwba wrote: post #69
ImprobableTruth wrote: post #57

Earlier quoted context omitted.

You've made one huge mistake: Davinci's $0.02 is not just per 1k tokens generated but also context tokens consumed . So if you generate 50 tokens per request with 1k context, the price is actually 20 times as large at $0.40 per 1k tokens generated - much less palatable, costing 3 times as much as the cloud hosted version of this. And that's not even taking into account the gigantic markup cloud services have.

Most of the computational cost of producing an output token is spent on consuming input tokens (including previous output tokens that are fed back in); only the final unembedding matrix could be eliminated if you don't care about the output logits for the context. So it's not correct to only modify OpenAI's prices to account for the ratio of context tokens to output tokens. Both of them get multiplied by 20 (if that'…

No, because they're already taking that into account.

>Metric: generation throughput (token/s) = number of the generated tokens / (time for processing prompts + time for generation).

(Though they're doing batching, so this is an unfair comparison. Would be interesting to get single batch speed.)

Re: Running large language models like ChatGPT on a single GPU

#79

Post by terabytest »

baobabKoodaa wrote: post #23
ml_basics wrote: post #11

Very cool. Worth mentioning though that the highlighted figures (1.12 tok/s for OPT-175B for "FlexGen with Compression") are for inputs of 512 tokens and outputs of 32 tokens. Since decoder-only transformer memory requirements scale with the square of sequence lengths, things would probably slow down significantly for very long sequences, which would be required for a back-and-forth conversation. Still though, until…

> Since decoder-only transformer memory requirements scale with the square of sequence lengths, things would probably slow down significantly for very long sequences, which would be required for a back-and-forth conversation. You can use tricks to keep the sequence length down even if the conversation goes on for a long time. For example, you can use the model to summarize the first n-1 lines of the conversation and…

This is very interesting. Could you please elaborate and maybe share links to articles if you know of any?

Re: Running large language models like ChatGPT on a single GPU

#80

Post by ipsum2 »

JoshCole wrote: post #28
humanistbot wrote: post #22

Earlier quoted context omitted.

This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.

No, it isn't astronomical. It is smaller than that. Still large, but not astronomical.

Have you tried training a large model before? If not, you're probably discounting how difficult and expensive it is.
Post reply on HN