Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

161–170 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#161

Post by underlines »

Relevant: Since LLaMA leaked on torrent, it has been converted to Huggingface weights and it has been quantisized to 8bit for less vram requirements.

A few days ago it has also been quantisized to 4bit and 3bit is coming. The quantization method they use is from the GPTQ paper ( https://arxiv.org/abs/2210.17323 ) which leads to almost no quality degradation compared to the 16bit weights.

4 bit weights:

Model, weight size, vram req.

LLaMA-7B, 3.5GB, 6GB

LLaMA-13B, 6.5GB, 10GB

LLaMA-30B, 15.8GB, 20GB

LLaMA-65B, 31.2GB, 40GB

Here is a good overall guide for Linux and Windows:

https://rentry.org/llama-tard-v2#bonus-4-4bit-llama-basic-se...

I also wrote a guide how to get the bitsandbytes library working on windows:

https://github.com/oobabooga/text-generation-webui/issues/14...

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#163

Post by nanidin »

adeon wrote: post #102
gpm wrote: post #92

Earlier quoted context omitted.

My code for this is very much not high quality, but I have a CPU + GPU + SSD combination: https://github.com/gmorenz/llama/tree/ssd Usage instructions in the commit message: https://github.com/facebookresearch/llama/commit/5be06e56056... At least with my hardware this runs at "[size of model]/[speed of SSD reads]" tokens per second, which (up to some possible further memory reduction so you can run larger batches at…

Yeah, it does seem like there's a fundamental limit how fast you can go even if you engineer the data juggling to perfection. My guess is that every loop through the transformer is going to have to visit every weight and if those weights cannot fit in your fastest memory, then it's going to have to spend time transferring data from SSD or whatever is lower in your memory hierarchy. The quantization used in the post l…

Perhaps there is an instance of Amdahl's law lurking the the midst?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#164

Post by lawrencechen »

Georgi just added support for all models (13B/33B/65B) [0]

LLaMA 65B can do ~2 tokens per second on my M1 Max / 64 gb ram [1]

[0] https://twitter.com/ggerganov/status/1634488664150487041 [1] https://twitter.com/lawrencecchen/status/1634507648824676353

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#165

Post by j45 »

Someone wrote: post #134
fwlr wrote: post #104

A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…

> All of the specs indicate an iPhone 14 Pro could achieve similar throughput to an M1 MacBook Pro. Battery capacity and thermals are different and might be problematic. The phone might throttle performance earlier. > it looks like we have all the pieces needed to make a ChatGPT-level assistant operate entirely through voice and run entirely on your phone. As a demo, yes, but would loading the model be fast enough fo…

Siri doesn’t seem as fast or responsive compared to Google assistant at times.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#168

Post by sebzim4500 »

RonnieOwnsLexus wrote: post #133
fwlr wrote: post #104

A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…

But wont it be that in real life no one would want to run a voice command which consumes lot of CPU and battery as opposed to making a network call to a service which has this model hosted ? Agreed that this can always be improved and hardware can get more efficient and better to but at the end of the day, would it ever be better then an API call ?

Ultimately, no amount of technology will ever beat the speed of light. Running locally will always have a lower latency floor.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#169

Post by MacsHeadroom »

amelius wrote: post #156
MacsHeadroom wrote: post #154

Earlier quoted context omitted.

Because that's not how machine learning models work. Machine learning as a field goes through a nearly complete revolution annually. Every new major model is a special snowflake of unique cases. Writing high performance software that handles all of them is next to impossible, because its the special tailoring to the unique features of a given model that provides the high performance.

That's not how I think it works. ML is a small number of operations applied to very large blocks of data, tensors. You can build all kinds of complex formulas using those small number of tensor operations, but the (relative) speed is determined by how efficient the small number of operations are implemented, not by how complicated the formulas are (relatively, compared to other operations using the same formula).

You're half right. First, tensor operations are only a small part of modern ML. Second, how you plug all those small operation together is where all the performance difference is had these days between implementations.

Different hardware have a variety of different small operations that do almost the same thing. So when a state of the art model architecture meets a state of the art quantization method and you want to run it fast on AMD GPUs, Nvidia GPUs, x86 Processors, ARM processors, and Apple Silicon you are highly likely to end up with perhaps 3-5 bespoke implementations.

This happens every few months in ML. Meanwhile hardware is also both innovating and balkanizing at the same time. Now we have Google Silicon, Huawei Silicon, and Intel Arc GPUs. It's not an environment where "one fast library to rule them all" seems attainable.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#170

Post by KVFinn »

lawrencechen wrote: post #164

Georgi just added support for all models (13B/33B/65B) [0] LLaMA 65B can do ~2 tokens per second on my M1 Max / 64 gb ram [1] [0] https://twitter.com/ggerganov/status/1634488664150487041 [1] https://twitter.com/lawrencecchen/status/1634507648824676353

Very cool. I've seen some people running 4-bit 65B on dual 3090s, but didn't notice a benchmark yet to compare.

It looks like this is regular 4-bit and not GPTQ 4-bit? It's possible there's quality loss but we'll have to test.

>4-bit quantization tends to come at a cost of substantial output quality losses. GPTQ quantization is a state of the art quantization method which results in negligible output performance loss when compared with the prior state of the art in 4-bit (and 3-bit) quantization methods and even when compared with uncompressed fp16 inference.

https://github.com/ggerganov/llama.cpp/issues/9

Post reply on HN