Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs on Apple Neural Engine (ANE)

github.com

Post reply on HN
Print view View original

41–50 of 132 posts

Re: Run LLMs on Apple Neural Engine (ANE)

#41

Post by kamranjon »

jorvi wrote: post #34
kamranjon wrote: post #13

Earlier quoted context omitted.

Whisper.cpp has a coreml option which gives 3x speed up over cpu only according to the docs: https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#c...

.. who is running LLMs on CPU instead of GPU or TPU/NPU

Actually that's a really good question, I hadn't considered that the comparison here is just CPU vs using Metal (CPU+GPU).

To answer the question though - I think this would be used for cases where you are building an app that wants to utilize a small AI model while at the same time having the GPU free to do graphics related things, which I'm guessing is why Apple stuck these into their hardware in the first place.

Here is an interesting comparison between the two from a whisper.cpp thread - ignoring startup times - the CPU+ANE seems about on par with CPU+GPU: https://github.com/ggml-org/whisper.cpp/pull/566#issuecommen...

Re: Run LLMs on Apple Neural Engine (ANE)

#42

Post by zozbot234 »

kamranjon wrote: post #37
zozbot234 wrote: post #24

Earlier quoted context omitted.

Some outdated information about bare-metal use of the ANE is available from the Whisper.cpp pull req: https://github.com/ggml-org/whisper.cpp/pull/1021 Even more outdated information at: https://github.com/eiln/ane/tree/33a61249d773f8f50c02ab0b9fe... In short, the early (M1/M2) versions of ANE are unlikely to be useful for modern LLM inference due to their seemingly exclusive focus on statically scheduled FP16 and IN…

I wouldn't say that they aren't useful for inference (there are pretty clear performance improvements even from the asahi effort you linked) - it's just that you have to convert the model ahead of time to be compatible with the ANE which is explained in the readme docs for whisper.cpp that I linked above. I would say though that this likely excludes them from being useful for training purposes.

Note that I was only commenting on modern quantized LLM's that basically avoid formats like FP16 or INT8, preferring lower precision wherever feasible. When in-memory model values must be padded to FP16/INT8 this slashes your effective use of memory bandwidth, which is what determines token generation speed. So the only feasible benefits are really in the prompt pre-processing phase, and even then only in lower power use compared to GPU, not really in higher speed.

Re: Run LLMs on Apple Neural Engine (ANE)

#44

Post by mr_toad »

lucasoshiro wrote: post #7
htk wrote: post #2

I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.

I'm not a ML guy, but when I needed to train a NN I thought that the my Mac's ANE would help. But actually, despite it being way easier to setup tensorflow + metal + M1 on Mac than to setup tensorflow + cuda + nvidia on Linux, the neural engine cores are not used. Not even for classification, which are their main purpose. I wouldn't say they are wasted silicon, but they are way less useful than what we expect

Does Apple care about third party use of the ANE? There are many iOS/iPadOS features that use it.

Re: Run LLMs on Apple Neural Engine (ANE)

#45

Post by echelon »

kamranjon wrote: post #13
behnamoh wrote: post #6

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…

Whisper.cpp has a coreml option which gives 3x speed up over cpu only according to the docs: https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#c...

> coreml option which gives 3x speed up over cpu

Which is still painfully slow. CoreML is not a real ML platform.

Re: Run LLMs on Apple Neural Engine (ANE)

#47

Post by kamranjon »

zozbot234 wrote: post #42
kamranjon wrote: post #37

Earlier quoted context omitted.

I wouldn't say that they aren't useful for inference (there are pretty clear performance improvements even from the asahi effort you linked) - it's just that you have to convert the model ahead of time to be compatible with the ANE which is explained in the readme docs for whisper.cpp that I linked above. I would say though that this likely excludes them from being useful for training purposes.

Note that I was only commenting on modern quantized LLM's that basically avoid formats like FP16 or INT8, preferring lower precision wherever feasible. When in-memory model values must be padded to FP16/INT8 this slashes your effective use of memory bandwidth, which is what determines token generation speed. So the only feasible benefits are really in the prompt pre-processing phase, and even then only in lower power…

That's really interesting! I didn't know that about the padding behavior here. I am interested to know which models this would include? I know Gemma 3 raw is bf16 - are you just talking about the quantized versions of these? Or are models being released purely as quantized versions these days? I know Google just released a QAT (Quantization Aware Training) model of Gemma 3 27b - but that base model was already released.

Re: Run LLMs on Apple Neural Engine (ANE)

#48

Post by conradev »

kamranjon wrote: post #37
zozbot234 wrote: post #24

Earlier quoted context omitted.

Some outdated information about bare-metal use of the ANE is available from the Whisper.cpp pull req: https://github.com/ggml-org/whisper.cpp/pull/1021 Even more outdated information at: https://github.com/eiln/ane/tree/33a61249d773f8f50c02ab0b9fe... In short, the early (M1/M2) versions of ANE are unlikely to be useful for modern LLM inference due to their seemingly exclusive focus on statically scheduled FP16 and IN…

I wouldn't say that they aren't useful for inference (there are pretty clear performance improvements even from the asahi effort you linked) - it's just that you have to convert the model ahead of time to be compatible with the ANE which is explained in the readme docs for whisper.cpp that I linked above. I would say though that this likely excludes them from being useful for training purposes.

My understanding is that model throughput is fundamentally limited at some point by the fact that the ANE is less wide than the GPU.

At that point, the ANE loses because you have to split the model into chunks and only one fits at a time.

Re: Run LLMs on Apple Neural Engine (ANE)

#49

Post by ljosifov »

behnamoh wrote: post #6

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…

This more than anything feels emblematic to me, that Apple executives are brain dead when it comes to software. AI seemingly being a step too far. (in the s/w direction.) While they could at some level grok classical s/w, NN-s are that Terra Incognita where Apple executives can not possibly follow in. It's just too strange and mysterious world for them to be able to effectively decide or execute on anything. Worked (20yrs ago) in an industrial R&D lab of a mostly h/w manufacturer for 3 years. It looked to me that the worlds of h/w and s/w, the mindsets, diverged pretty quickly on all important considerations of "what should happen next".

Re: Run LLMs on Apple Neural Engine (ANE)

#50

Post by zozbot234 »

kamranjon wrote: post #47
zozbot234 wrote: post #42

Earlier quoted context omitted.

Note that I was only commenting on modern quantized LLM's that basically avoid formats like FP16 or INT8, preferring lower precision wherever feasible. When in-memory model values must be padded to FP16/INT8 this slashes your effective use of memory bandwidth, which is what determines token generation speed. So the only feasible benefits are really in the prompt pre-processing phase, and even then only in lower power…

That's really interesting! I didn't know that about the padding behavior here. I am interested to know which models this would include? I know Gemma 3 raw is bf16 - are you just talking about the quantized versions of these? Or are models being released purely as quantized versions these days? I know Google just released a QAT (Quantization Aware Training) model of Gemma 3 27b - but that base model was already releas…

Models may be released as unquantized (and even then they are gradually shifting towards lower precisions over time), but most people are going to be running them in a quantized version simply because that gives you the best bang for your buck (you can fit more interesting models on the same hardware). Of course this is strictly about local LLM inference, though one may reasonably assume that the big players are also doing something similar.
Post reply on HN