Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs on Apple Neural Engine (ANE)

github.com

Post reply on HN
Print view View original

11–20 of 132 posts

Re: Run LLMs on Apple Neural Engine (ANE)

#11

Post by [deleted] »

bigyabai wrote: post #5
htk wrote: post #2

I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.

If you did that, you'd stumble into the Apple GPU's lack of tensor acceleration hardware. For an Nvidia-like experience you'd have to re-architecture the GPU to subsume the NPU's role, and if that was easy then everyone would have done it by now.

[deleted]

Re: Run LLMs on Apple Neural Engine (ANE)

#12

Post by kamranjon »

486sx33 wrote: post #10
kamranjon wrote: post #4

I am curious if anyone knows if the neural cores in apple silicon based machines are at all useful in training? I’ve been using the MLX framework but haven’t seen them mentioned anywhere so I’m just wondering if they are only useful for inference? I know whisper.cpp takes advantage of them in the inference context. Edit: I changed llama.cpp to whisper.cpp - I didn’t realize that llama.cpp doesn’t have a coreml option…

Maybe a quick side shift - What the heck are apples neural cores good for ? Used for ? Use cases ?

They are definitely good for local inference as evidenced by the pretty amazing performance increase on Apple silicon when used with whisper.cpp, maybe other frameworks that utilize coreml? I think they’re sorta purpose built for doing matrix math.

Re: Run LLMs on Apple Neural Engine (ANE)

#13

Post by kamranjon »

behnamoh wrote: post #6

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…

Whisper.cpp has a coreml option which gives 3x speed up over cpu only according to the docs: https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#c...

Re: Run LLMs on Apple Neural Engine (ANE)

#14

Post by cowmix »

This sorta reminds me of the lie that was pushed when the Snapdragon X laptops were being released last year. Qualcomm implied the NPU would be used for LLMs — and I bought into the BS without looking into it. I still use a Snapdragon laptop as my daily driver (it's fine) but for running models locally, it's still a joke. Despite Qualcomm's claims about running 13B parameter models, software like LM Studio only runs on CPU with NPU support merely "planned for future updates." XDA The NPU isn't even faster than the CPU for LLMs — it's just more power-efficient for small models, not the big ones people actually want to run. Their GPUs aren't much better for this purpose either. The only hope for LLMs is the Vulkan support on the Snapdragon X — which still is half-baked.

Re: Run LLMs on Apple Neural Engine (ANE)

#15

Post by nullpoint420 »

cowmix wrote: post #14

This sorta reminds me of the lie that was pushed when the Snapdragon X laptops were being released last year. Qualcomm implied the NPU would be used for LLMs — and I bought into the BS without looking into it. I still use a Snapdragon laptop as my daily driver (it's fine) but for running models locally, it's still a joke. Despite Qualcomm's claims about running 13B parameter models, software like LM Studio only runs…

If you don’t mind me asking, what OS do you use on it?

Re: Run LLMs on Apple Neural Engine (ANE)

#16

Post by ks2048 »

behnamoh wrote: post #6

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…

One update from Apple Research since that one was "Deploying Attention-Based Vision Transformers to Apple Neural Engine" (but not clear the relationship. It doesn't build on ane_transformers, but maybe a sister project for vision?)

blog: https://machinelearning.apple.com/research/vision-transforme...

github: https://github.com/apple/ml-vision-transformers-ane

Re: Run LLMs on Apple Neural Engine (ANE)

#17

Post by 1W6MIC49CYX9GAP »

behnamoh wrote: post #6

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…

Based on the graphs "up to 10 times faster" compares before/after flash attention

Re: Run LLMs on Apple Neural Engine (ANE)

#18

Post by ks2048 »

htk wrote: post #2

I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.

At least one link/benchmark I saw said the ANE can be 7x faster than GPU (Metal / MPS),

https://discuss.pytorch.org/t/apple-neural-engine-ane-instea...

It seems intuitive that if they design hardware very specifically for these applications (beyond just fast matmuls on a GPU), they could squeeze out more performance.

Re: Run LLMs on Apple Neural Engine (ANE)

#19

Post by 1W6MIC49CYX9GAP »

htk wrote: post #2

I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.

You're completely right, if you already have a GPU in a system adding tensor cores to it gives you better performance per area.

GPU + dedicated AI HW is virtually always the wrong approach compared to GPU+ tensor cores

Re: Run LLMs on Apple Neural Engine (ANE)

#20

Post by xiphias2 »

htk wrote: post #2

I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.

I guess it's a hard choice as it's 5x more energy efficient than GPU because it uses systolic array.

For laptops, 2x GPU cores would make more sense, for phones/tablets, energy efficiency is everything.

Post reply on HN