Live data from Hacker News

Run LLMs on Apple Neural Engine (ANE)

131–132 of 132 posts

Re: Run LLMs on Apple Neural Engine (ANE)

#131

Post by anemll » Wed, May 07, 2025, 4:14 PM UTC

SparkyMcUnicorn wrote: post #108

anemll wrote: post #106

Earlier quoted context omitted.

What hardware are you on? Most models are memory bandwidth limited. ANE was limited to 64GB/s prior to M3 Max or M4 pro. If you are on M1, GPU will be significantly faster for 3-8B models due to memory bandwidth rather then ANE capabilities.

M4 Max with 128GB of memory.

M4 max should work at 120GB for ANE and 500+ for GPU. So GPU will be 3-4 times faster for anything over 1-3B. ANE is likely as fast for prefill due to higher FLOPs

Re: Run LLMs on Apple Neural Engine (ANE)

#132

Post by anemll » Wed, May 07, 2025, 4:15 PM UTC

sunpazed wrote: post #115

The key benefit is significant lower power usage. Benchmarked llama3.2-1B on my machines; M1 Max (47t/s, ~1.8 watts), M4 Pro (62t/s, ~2.8 watts). The GPU is twice as fast (even faster on the Max), but draws much more power (~20 watts) vs the ANE. Also the ANE models are limited to 512 tokens of context, so unlikely yet to use these in production.

We can ran 2000 or 4000 context with ANE

Post reply on HN