Run LLMs on Apple Neural Engine (ANE)
github.com
Run LLMs on Apple Neural Engine (ANE)
1–10 of 132 posts
Re: Run LLMs on Apple Neural Engine (ANE)
#2I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.
Re: Run LLMs on Apple Neural Engine (ANE)
#3I'm trying to figure out what the secret sauce for this is. It depends on https://github.com/apple/coremltools - is that the key trick or are there other important techniques going on here?
Re: Run LLMs on Apple Neural Engine (ANE)
#4I am curious if anyone knows if the neural cores in apple silicon based machines are at all useful in training? I’ve been using the MLX framework but haven’t seen them mentioned anywhere so I’m just wondering if they are only useful for inference? I know whisper.cpp takes advantage of them in the inference context.
Edit: I changed llama.cpp to whisper.cpp - I didn’t realize that llama.cpp doesn’t have a coreml option like whisper.cpp does.
Re: Run LLMs on Apple Neural Engine (ANE)
#5I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.
If you did that, you'd stumble into the Apple GPU's lack of tensor acceleration hardware. For an Nvidia-like experience you'd have to re-architecture the GPU to subsume the NPU's role, and if that was easy then everyone would have done it by now.
Re: Run LLMs on Apple Neural Engine (ANE)
#6I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers
They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations."
AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0].
What's weird is that MLX is made by Apple and yet, they can't support ANE given its closed-source API! [1]
[0]: https://github.com/ggml-org/llama.cpp/issues/10453
[1]: https://github.com/ml-explore/mlx/issues/18#issuecomment-184...
Re: Run LLMs on Apple Neural Engine (ANE)
#7I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.
I'm not a ML guy, but when I needed to train a NN I thought that the my Mac's ANE would help. But actually, despite it being way easier to setup tensorflow + metal + M1 on Mac than to setup tensorflow + cuda + nvidia on Linux, the neural engine cores are not used. Not even for classification, which are their main purpose. I wouldn't say they are wasted silicon, but they are way less useful than what we expect
Re: Run LLMs on Apple Neural Engine (ANE)
#8I am curious if anyone knows if the neural cores in apple silicon based machines are at all useful in training? I’ve been using the MLX framework but haven’t seen them mentioned anywhere so I’m just wondering if they are only useful for inference? I know whisper.cpp takes advantage of them in the inference context. Edit: I changed llama.cpp to whisper.cpp - I didn’t realize that llama.cpp doesn’t have a coreml option…
Well, the TensorFlow port to metal was written by Apple, and it doens't use ANE. If even they have chosen to use only GPU probably the ANE wouldn't help in training. I also heard that the ANE is way less powerful than Apple Silicon's GPU, but I don't have numbers
Re: Run LLMs on Apple Neural Engine (ANE)
#9I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.
[deleted]
Re: Run LLMs on Apple Neural Engine (ANE)
#10I am curious if anyone knows if the neural cores in apple silicon based machines are at all useful in training? I’ve been using the MLX framework but haven’t seen them mentioned anywhere so I’m just wondering if they are only useful for inference? I know whisper.cpp takes advantage of them in the inference context. Edit: I changed llama.cpp to whisper.cpp - I didn’t realize that llama.cpp doesn’t have a coreml option…
Maybe a quick side shift -
What the heck are apples neural cores good for ? Used for ? Use cases ?