Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs on Apple Neural Engine (ANE)

github.com

Post reply on HN
Print view View original

31–40 of 132 posts

Re: Run LLMs on Apple Neural Engine (ANE)

#31

Post by noahbp »

behnamoh wrote: post #6

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…

Onnxruntime supports CoreML, though if my experience with converting an embedding model to CoreML using Apple's CoreML conversion tool is similar to the ORT maintainers', I can see why it would be unmaintained.

It took multiple tries to get the model to convert at all to the mlpackage format, and then a lot of experimenting to get it to run on the ANE instead of the GPU, only to discover that constant reshaping was killing any performance benefit (either you have a fixed multiplication size or don't bother), and even at a fixed size and using the attention mask, its operations were slower than saturating the GPU with large batches.

I discovered an issue where using the newer iOS 18 standard would cause the model conversion to break, and put an issue in on their GitHub, including an example repository for easy replication. I got a response quickly, but almost a year later, the bug is still unfixed.

Even when George Hotz attempted to hack it to use it without Apple's really bad and unmaintained CoreML library, he gave up because it was impossible without breaking some pretty core OS features (certificate signing IIRC).

The ANE/CoreML is just not serious at all about making their hardware usable at all. Even Apple's internal MLX team can't crack that nut.

Re: Run LLMs on Apple Neural Engine (ANE)

#33

Post by brigade »

htk wrote: post #2

I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.

Eyeballing 3rd party annotated die shots [1], it’s about the size of two GPU cores, but achieves 15.8 tflops. Which is more than the reported 14.7 tflops of the 32-core GPU in the binned M4 Max.

[1] https://vengineer.hatenablog.com/entry/2024/10/13/080000

Re: Run LLMs on Apple Neural Engine (ANE)

#34

Post by jorvi »

kamranjon wrote: post #13
behnamoh wrote: post #6

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…

Whisper.cpp has a coreml option which gives 3x speed up over cpu only according to the docs: https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#c...

.. who is running LLMs on CPU instead of GPU or TPU/NPU

Re: Run LLMs on Apple Neural Engine (ANE)

#35

Post by mikaraento »

captainregex wrote: post #21
cowmix wrote: post #14

This sorta reminds me of the lie that was pushed when the Snapdragon X laptops were being released last year. Qualcomm implied the NPU would be used for LLMs — and I bought into the BS without looking into it. I still use a Snapdragon laptop as my daily driver (it's fine) but for running models locally, it's still a joke. Despite Qualcomm's claims about running 13B parameter models, software like LM Studio only runs…

AnythingLLM uses NPU

Could you provide a pointer to docs for this? It wasn't obvious from an initial read of their docs.

Re: Run LLMs on Apple Neural Engine (ANE)

#37

Post by kamranjon »

zozbot234 wrote: post #24
kamranjon wrote: post #13

Earlier quoted context omitted.

Whisper.cpp has a coreml option which gives 3x speed up over cpu only according to the docs: https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#c...

Some outdated information about bare-metal use of the ANE is available from the Whisper.cpp pull req: https://github.com/ggml-org/whisper.cpp/pull/1021 Even more outdated information at: https://github.com/eiln/ane/tree/33a61249d773f8f50c02ab0b9fe... In short, the early (M1/M2) versions of ANE are unlikely to be useful for modern LLM inference due to their seemingly exclusive focus on statically scheduled FP16 and IN…

I wouldn't say that they aren't useful for inference (there are pretty clear performance improvements even from the asahi effort you linked) - it's just that you have to convert the model ahead of time to be compatible with the ANE which is explained in the readme docs for whisper.cpp that I linked above.

I would say though that this likely excludes them from being useful for training purposes.

Re: Run LLMs on Apple Neural Engine (ANE)

#38

Post by freehorse »

486sx33 wrote: post #10
kamranjon wrote: post #4

I am curious if anyone knows if the neural cores in apple silicon based machines are at all useful in training? I’ve been using the MLX framework but haven’t seen them mentioned anywhere so I’m just wondering if they are only useful for inference? I know whisper.cpp takes advantage of them in the inference context. Edit: I changed llama.cpp to whisper.cpp - I didn’t realize that llama.cpp doesn’t have a coreml option…

Maybe a quick side shift - What the heck are apples neural cores good for ? Used for ? Use cases ?

In my understanding, they are intended to be utilized by various apps (and Apple intelligence) for performing different machine learning tasks (inference), in a manner that is unobtrusive to the user and the rest of the system. While a GPU would theoretically be more performant, it could potentially lead to excessive energy consumption, temperature rise, fan noise, and other factors that may not be desirable when performing basic tasks like OCR when viewing an image within an application.

Currently, it is for example used through the "Vision Framework" eg for OCR tasks (for instance, when previewing an image in macOS, it performs OCR in the background using the ANE). Additionally, they are utilized by certain apple intelligence features that are executed locally (eg when I asked writing tools to rewrite this comment, I saw a spike in ANE usage).

They can also be used for diffusion image models (through core ml, diffusers has a nice frontend for that) but my understanding is that they are primarily for "light" ML tasks within an application rather than running larger models (though that's also possible, but they are gonna probably run slower than in gpu).

Re: Run LLMs on Apple Neural Engine (ANE)

#39

Post by fmajid »

daedrdev wrote: post #26

Apple is a competitive choice simply because their unified memory allows you to get enough ram that would take multiple Gpus to have enough space to run larger models.

Yes, but their refusal to open up the ANE to third-party models negates that. You can get (or will be able to very soon) a Strix Halo Ryzen AI Max+ 395 able to access 96GB of unified RAM (on a 128GB system) for well under half what you'd pay for an equivalent M4 system from Apple.

Re: Run LLMs on Apple Neural Engine (ANE)

#40

Post by thot_experiment »

jorvi wrote: post #34
kamranjon wrote: post #13

Earlier quoted context omitted.

Whisper.cpp has a coreml option which gives 3x speed up over cpu only according to the docs: https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#c...

.. who is running LLMs on CPU instead of GPU or TPU/NPU

[flagged]
Post reply on HN