Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

91–100 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#91

Post by kamranjon »

lynguist wrote: post #89
eigenvalue wrote: post #83

This is so awesome and exciting. I have an M1 iMac and it was trivially easy to get this working and generating text. And the performance is VERY impressive, especially considering that it's not even using any of the built in "neural compute" stuff. Also, the model seems like it doesn't have any political correctness conditioning based on some of the completions it has given me on controversial prompts. I can't wait…

It is using the built-in neural accelerators, that’s why it’s fast, that’s why it’s only supported on Macs so far. The code makes use of official Apple APIs which delegate the necessary BLAS calls to the available hardware.

Confusingly there are 2 mechanisms to do matrix operations on the new apple hardware - AMX (https://github.com/corsix/amx) - and the ANE (apple neural engine) - which is enabled by CoreML. This code does not run on the neural engine but the author has a branch for his whisper.cpp project which uses it here: https://github.com/ggerganov/whisper.cpp/pull/566 - so it may not be long before we see it applied here as well. All of this is to say that it actually could get significantly faster if some of this work was able to be handed to the ANE with CoreML.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#92

Post by gpm »

adeon wrote: post #81

If you are interested in implementing LLaMA yourself or learning, I noticed that the reference code by Facebook is one of the cleaner, easier to read ML code I've seen in a while. https://github.com/facebookresearch/llama/blob/main/llama/mo... It's about 200 lines long. You probably do need a bit of knowledge to understand what you are reading but I was pleasantly surprised. For example in comparison, StableDiffusion…

My code for this is very much not high quality, but I have a CPU + GPU + SSD combination: https://github.com/gmorenz/llama/tree/ssd

Usage instructions in the commit message: https://github.com/facebookresearch/llama/commit/5be06e56056...

At least with my hardware this runs at "[size of model]/[speed of SSD reads]" tokens per second, which (up to some possible further memory reduction so you can run larger batches at once on the same GPU) is a good as it gets when you need to read the whole model from disk each token.

At a 125GB and a 2MB/s read (largest model, what I get from my ssd) that's 60 seconds per token (1 day per 1440 words), which isn't exactly practical. Which is really the issue here, if you need to stream the model from an SSD because you don't have enough RAM, it is just a fundamentally slow process.

You could probably optimize quite a bit for batch throughput if you're ok with the latency though.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#93

Post by lynguist »

kamranjon wrote: post #91
lynguist wrote: post #89

Earlier quoted context omitted.

It is using the built-in neural accelerators, that’s why it’s fast, that’s why it’s only supported on Macs so far. The code makes use of official Apple APIs which delegate the necessary BLAS calls to the available hardware.

Confusingly there are 2 mechanisms to do matrix operations on the new apple hardware - AMX ( https://github.com/corsix/amx ) - and the ANE (apple neural engine) - which is enabled by CoreML. This code does not run on the neural engine but the author has a branch for his whisper.cpp project which uses it here: https://github.com/ggerganov/whisper.cpp/pull/566 - so it may not be long before we see it applied here as we…

You’re right! I wrote it too fast without thinking!

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#94

Post by capableweb »

lostmsu wrote: post #69
xiphias2 wrote: post #5

George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch: https://github.com/geohot/tinygrad/tree/llama The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.

It's not tinygrad really, it is PyTorch.

Unfamiliar with the domain, but is it? Looking through the changes (https://github.com/geohot/tinygrad/compare/master...llama) it seems to not mention pytorch at all but tinygrad + numpy.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#95

Post by zamnos »

adeon wrote: post #81

If you are interested in implementing LLaMA yourself or learning, I noticed that the reference code by Facebook is one of the cleaner, easier to read ML code I've seen in a while. https://github.com/facebookresearch/llama/blob/main/llama/mo... It's about 200 lines long. You probably do need a bit of knowledge to understand what you are reading but I was pleasantly surprised. For example in comparison, StableDiffusion…

tinygrad by geohot, also linked on this thread, has similar properties good for learning - it's a couple hundred LoC to integrate LLaMA support

https://github.com/geohot/tinygrad/tree/llama

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#96

Post by lynguist »

kiratp wrote: post #66
0xbadc0de5 wrote: post #24

That's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig. Coupled with the leaked Bing prompt and text-ge…

Apple Silicon uses unified memory so laptops have up to 64GB of VRAM and the Max sitio can have up to 128 GB. That makes them uniquely “powerful” for inference with large models.

Nowadays up to 96GB on the laptops.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#98

Post by sillysaurusx »

adeon wrote: post #81

If you are interested in implementing LLaMA yourself or learning, I noticed that the reference code by Facebook is one of the cleaner, easier to read ML code I've seen in a while. https://github.com/facebookresearch/llama/blob/main/llama/mo... It's about 200 lines long. You probably do need a bit of knowledge to understand what you are reading but I was pleasantly surprised. For example in comparison, StableDiffusion…

Just don’t copy their sampler. It’s trashcan-tier. Bad defaults and no repetition penalty.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#99

Post by sillysaurusx »

eigenvalue wrote: post #83

This is so awesome and exciting. I have an M1 iMac and it was trivially easy to get this working and generating text. And the performance is VERY impressive, especially considering that it's not even using any of the built in "neural compute" stuff. Also, the model seems like it doesn't have any political correctness conditioning based on some of the completions it has given me on controversial prompts. I can't wait…

Repetition penalty is a matter of, generate a token, then multiply that logit by the penalty. (If the logit is negative, divide instead of multiply.)

https://github.com/shawwn/llama has an implementation (check the commit history).

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#100

Post by sillysaurusx »

lxe wrote: post #35

I'm running 4-bit quantized llamas on torch/cuda with https://github.com/qwopqwop200/GPTQ-for-LLaMa , and I'm seeing significant tokens/second perf degradation compared to 8-bit bitsandbytes mode. I'm very new to this, and understand very little detail, but I thought it would be faster?

Eh, I’d expect it to be slower if anything. Think about it like this. If you write an image renderer, bitmap would be the fastest, because it’s already decompressed. 4-bit quantization is a compression algorithm.

It depends on the details of memory bandwidth vs compute though.

Post reply on HN