Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs on Apple Neural Engine (ANE)

github.com

Post reply on HN
Print view View original

51–60 of 132 posts

Re: Run LLMs on Apple Neural Engine (ANE)

#51

Post by sroussey »

mr_toad wrote: post #44
lucasoshiro wrote: post #7

Earlier quoted context omitted.

I'm not a ML guy, but when I needed to train a NN I thought that the my Mac's ANE would help. But actually, despite it being way easier to setup tensorflow + metal + M1 on Mac than to setup tensorflow + cuda + nvidia on Linux, the neural engine cores are not used. Not even for classification, which are their main purpose. I wouldn't say they are wasted silicon, but they are way less useful than what we expect

Does Apple care about third party use of the ANE? There are many iOS/iPadOS features that use it.

Not really. Apple software uses the neural engine all over the place, but rarely do others. Maybe this will change [1]

There was a guy using it for live video transformations and it almost caused the phones to “melt”. [2]

[1] https://machinelearning.apple.com/research/neural-engine-tra...

[2] https://x.com/mattmireles/status/1916874296460456089

Re: Run LLMs on Apple Neural Engine (ANE)

#53

Post by conradev »

kamranjon wrote: post #41
jorvi wrote: post #34

Earlier quoted context omitted.

.. who is running LLMs on CPU instead of GPU or TPU/NPU

Actually that's a really good question, I hadn't considered that the comparison here is just CPU vs using Metal (CPU+GPU). To answer the question though - I think this would be used for cases where you are building an app that wants to utilize a small AI model while at the same time having the GPU free to do graphics related things, which I'm guessing is why Apple stuck these into their hardware in the first place. H…

It essentially never makes sense to run on the CPU and you will only ever see enthusiasts doing it.

Yes, hammering the GPU too hard can affect the display server, but no, switching to the CPU is not a good alternative

Re: Run LLMs on Apple Neural Engine (ANE)

#54

Post by smpanaro »

behnamoh wrote: post #6

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…

Not a public follow-up but the iOS 17 speech-to-text model has a clever approach to KV caching that works within the ANE’s constraints (fixed size inputs).

I wrote about it here[0] but the gist is you can have a fixed size cache and slide it in chunks with each inference. Not as efficient as a cache that grows by one each time of course.

[0]: https://stephenpanaro.com/blog/inside-apples-2023-transforme...

Re: Run LLMs on Apple Neural Engine (ANE)

#56

Post by kamranjon »

conradev wrote: post #53
kamranjon wrote: post #41

Earlier quoted context omitted.

Actually that's a really good question, I hadn't considered that the comparison here is just CPU vs using Metal (CPU+GPU). To answer the question though - I think this would be used for cases where you are building an app that wants to utilize a small AI model while at the same time having the GPU free to do graphics related things, which I'm guessing is why Apple stuck these into their hardware in the first place. H…

It essentially never makes sense to run on the CPU and you will only ever see enthusiasts doing it. Yes, hammering the GPU too hard can affect the display server, but no, switching to the CPU is not a good alternative

Not switching to the CPU - switching to the ANE (Neural Cores) - if you read the research papers Apple has released - the example I gave is pretty much how it's being used - small image classification models running on the ANE, alongside a graphics app that needs the GPU to be free.

Re: Run LLMs on Apple Neural Engine (ANE)

#57

Post by echelon »

kamranjon wrote: post #43
thot_experiment wrote: post #40

Earlier quoted context omitted.

[flagged]

Pretty sure they're using the 80 GPU cores available in that case.

And that still performs worse than entry-level Nvidia gaming cards.

Apple isn't serious about AI and needs to figure their AI story out. Every other big tech company is doing something about it.

Re: Run LLMs on Apple Neural Engine (ANE)

#58

Post by zozbot234 »

gitroom wrote: post #55

Man, Apple's tight grip on ANE is kinda nuts - would love to see the day they let folks get real hands-on. you ever think companies hold stuff back just to keep control, or is there actually some big tech reason for it?

People keep saying this but I'm not seeing the big difference with other NPU varieties. Either way we're still talking about very experimental stuff that also tends to be hardwired towards some pre-determined use case. So I'm not surprised that people are running into problems while trying to make these more broadly useful.

Re: Run LLMs on Apple Neural Engine (ANE)

#59

Post by smpanaro »

conradev wrote: post #48
kamranjon wrote: post #37

Earlier quoted context omitted.

I wouldn't say that they aren't useful for inference (there are pretty clear performance improvements even from the asahi effort you linked) - it's just that you have to convert the model ahead of time to be compatible with the ANE which is explained in the readme docs for whisper.cpp that I linked above. I would say though that this likely excludes them from being useful for training purposes.

My understanding is that model throughput is fundamentally limited at some point by the fact that the ANE is less wide than the GPU. At that point, the ANE loses because you have to split the model into chunks and only one fits at a time.

What do you mean by less wide? The main bottleneck for transformers is memory bandwidth. ANE has a much lower ceiling than CPU/GPU (yes, despite unified memory).

Chunking is actually beneficial as long as all the chunks can fit into the ANE’s cache. It speeds up compilation for large network graphs and cached loads are negligible cost. On M1 the cache limit is 3-4GB, but it is higher on M2+.

Re: Run LLMs on Apple Neural Engine (ANE)

#60

Post by voidspark »

thot_experiment wrote: post #40
jorvi wrote: post #34

Earlier quoted context omitted.

.. who is running LLMs on CPU instead of GPU or TPU/NPU

[flagged]

M3 Ultra has a big GPU with 819 GB/sec bandwidth.

LLM performance is twice as fast as RTX 5090

https://creativestrategies.com/mac-studio-m3-ultra-ai-workst...

Post reply on HN