Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

241–250 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#241

Post by DennisAleynikov »

fwlr wrote: post #104

A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…

If I may, this library runs LLaMA on CPU. There is no way to run it on the Neural Engine yet.

The optimization in this case only seems to refer to the 4bit model loading method (to be friendlier to the arm64 CPU)

GeoHot has tinygrad running LLaMA on Metal (but only the 7B model) that's the closest I've seen to taking advantage of apple silicon.

Neural Engine implementation would be awesome

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#242

Post by brookst »

sebzim4500 wrote: post #168
RonnieOwnsLexus wrote: post #133

Earlier quoted context omitted.

But wont it be that in real life no one would want to run a voice command which consumes lot of CPU and battery as opposed to making a network call to a service which has this model hosted ? Agreed that this can always be improved and hardware can get more efficient and better to but at the end of the day, would it ever be better then an API call ?

Ultimately, no amount of technology will ever beat the speed of light. Running locally will always have a lower latency floor.

Theoretically yes. But in the real world, no.

Simple thought experiment: you want to know how many tons of copper are mined in the US each year. Lowest possible latency is calculating this in your head, most likely using data you don’t have. Looking it up online is a lot, lot faster.

In some far future world maybe every transistor will include the sum total of human knowledge up to the nanosecond, but that’s a pretty far future. There are many things where running locally means a higher latency floor.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#243

Post by nl »

minxomat wrote: post #181
MacsHeadroom wrote: post #171

Earlier quoted context omitted.

LLaMA it doesn't require any system RAM to run. It requires some very minimal system RAM to load the model into VRAM and to compile the 4bit quantized weights. But if you use pre-quantized weights (get them from HuggingFace or a friend) then all you really need is ~32GB of VRAM and maybe around 2GB of system RAM for 65B. (It's 30B which needs 20GB of VRAM.)

The full use case includes quantisation, which the repo points out uses a large amount of system RAM. Of course that’s not required if you skip that step.

Quantisation is a once off process. I suspect most people who don't have access to a machine with enough RAM and don't want to use the pre-quantized version can afford the $20 to hire a big cloud server for an day.

Or it is probably possible to make it work slowly using a swapfile on Linux.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#244

Post by brookst »

KVFinn wrote: post #172
recuter wrote: post #86

Earlier quoted context omitted.

That's a pretty important just. They chose to go down this path at this time and shipped, now I can run the 64B Llama on a widely available $3,999 prosumer device. How much would a PC that can do that currently cost me and can I have it by tomorrow?

>How much would a PC that can do that currently cost me and can I have it by tomorrow? At the moment, seems like Apple has an edge here. On PC for single GPU you need an NVIDIA A40, which used prices for is about $2500, and not at retail stores. If you don't mind having two GPUs then two $800 3090 GPUs works, but that's a workstation build you'll have to order from Puget or something. That's probably faster than Appl…

…and for models that require 64GB of VRAM? 120GB of VRAM?

You can get a 128GB UMA mac for less than a single 48GB a100, let alone a single 96GB a100.

I think Apple got incredibly lucky here, but I don’t see how the PC world catches them any time soon. We’ve all known that UMA is theoretically better for ages, but Apple’s timing couldn’t be better. And scale economies mean they can sell the same chip to people who need 100GB of system RAM and people who need 100GB of VRAM.

If they can get their GPU / neural performance up and sort out their terrible relationship with academic research, they could snipe ML away from nvidia. It seems very unlikely, but it’s kind of stunning that it’s even in the realm of possibility.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#245

Post by DennisAleynikov »

Nevermark wrote: post #88
recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

Given that the RAM is directly accessible by CPU, GPU and DPU/Neural cores, it really is premium RAM. Apple's visionary hardware team has finally caught up with Apple's visionary high RAM prices! :)

Also um... Idk if anyone cares but apple didn't do anything to make unified memory.

All arm chips do this lmao

Apple just has the best known arm chips with the highest mobile performance (yes faster server arm chips exist too)

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#246

Post by DennisAleynikov »

xiphias2 wrote: post #114
BoberMod wrote: post #63

Earlier quoted context omitted.

There is also a gpu-acelerated fork of the original repo https://github.com/remixer-dec/llama-mps

> For 7B model, it always goes above 32gb of RAM, That's double of what Tinygrad uses

Tinygrad is using openCL right?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#247

Post by fwlr »

DennisAleynikov wrote: post #241
fwlr wrote: post #104

A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…

If I may, this library runs LLaMA on CPU. There is no way to run it on the Neural Engine yet. The optimization in this case only seems to refer to the 4bit model loading method (to be friendlier to the arm64 CPU) GeoHot has tinygrad running LLaMA on Metal (but only the 7B model) that's the closest I've seen to taking advantage of apple silicon. Neural Engine implementation would be awesome

Oh shit, I took a closer look and you’re right. The repo was also helpfully updated with a note to this effect: “The Accelerate framework is actually currently unused since I found that for tensor shapes typical for the Decoder, there is no benefit compared to the ARM_NEON intrinsics implementation. Of course, it's possible that I simlpy don't know how to utilize it properly. But in any case, you can even disable it with LLAMA_NO_ACCELERATE=1 make and the performance will be the same, since no BLAS calls are invoked by the current implementation”.

No Joi in my pocket just yet :(

Because of this I re-checked my claims about the Whisper speed up from the Neural Engine and that does look legit, 6x at least. So the Neural Engine does have the chops for this workload, it just isn’t being used in this repo. It may not be LLaMA, but I sure hope someone gets an LLM running on the ANE sooner rather than later.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#248

Post by underlines »

zabzonk wrote: post #84
zabzonk wrote: post #54

Earlier quoted context omitted.

doesn't say what to do with cream. sprinkling with cheese is a bit wrong. carbonara sauce is more difficult to do than you (or an AI) might think.

well, does it say what to do with cream?

cream stays far far away from carbonara. otherwise it's not carbonara.

carbonara sauce is simply pecorino or parmigiano cheese mixed with eggs or just yolks and pepper and guanciale or pancetta.

NO CREAM, NO MILK, NO HAM, NO BACON! basta! /endofrant

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#250

Post by ar9av »

fwlr wrote: post #104

A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…

the potential drawbacks of relying entirely on voice-operated assistants like ChatGPT. There are concerns around privacy and the use of personal data, as well as the potential for bias and inaccuracies in the responses generated by these models. It's important to strike a balance between the convenience and benefits of these technologies and the potential risks and limitations they bring. Nonetheless, the advancements being made in this field are impressive and it will be interesting to see how they develop in the future.
Post reply on HN