Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs on Apple Neural Engine (ANE)

github.com

Post reply on HN
Print view View original

81–90 of 132 posts

Re: Run LLMs on Apple Neural Engine (ANE)

#81

Post by brigade »

Archit3ch wrote: post #78
brigade wrote: post #33

Earlier quoted context omitted.

Eyeballing 3rd party annotated die shots [1], it’s about the size of two GPU cores, but achieves 15.8 tflops. Which is more than the reported 14.7 tflops of the 32-core GPU in the binned M4 Max. [1] https://vengineer.hatenablog.com/entry/2024/10/13/080000

Not really. That's 15.8 fp16 ops compared to 14.7 fp32 ops (that are actually useful outside AI). It would be interesting to see if you can configure the ANE to recover fp32 precision at lower throughput [1]. [1] https://arxiv.org/abs/2203.03341

Apple GPUs run fp16 at the same rate as fp32 except on phones, so it is comparable for ML. No one runs inference from fp32 weights.

But the point was about area efficiency

Re: Run LLMs on Apple Neural Engine (ANE)

#82

Post by yjftsjthsd-h »

jorvi wrote: post #34
kamranjon wrote: post #13

Earlier quoted context omitted.

Whisper.cpp has a coreml option which gives 3x speed up over cpu only according to the docs: https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#c...

.. who is running LLMs on CPU instead of GPU or TPU/NPU

Not all of us own GPUs worth using. Now, among people using macs... Maybe if you had a hardware failure?

Re: Run LLMs on Apple Neural Engine (ANE)

#83

Post by wtallis »

zozbot234 wrote: post #74
wtallis wrote: post #70

Earlier quoted context omitted.

True; everybody's NPU hardware is afflicted by awkward hardware and software constraints that don't come close to keeping pace with the rapidly-shifting interests of ML researchers. To some degree, that's an unavoidable consequence of how long it takes to design and ship specialized hardware with a supporting software stack. By contrast, ML research is moving way faster because they hardly ever ship anything product-…

> It'll be very interesting to see how this space matures over the next several years, and whether the niche of specialized low-power NPUs survives in PCs or if NVIDIA's approach of only using the GPU wins out. GPU's are gaining their own kinds of specialized blocks such as matrix/tensor compute units, or BVH acceleration for ray-tracing (that may or may not turn out to be useful for other stuff). So I'm not sure tha…

> a specialized low-power unit in an iGPU is going to be practically indistinguishable from a NPU, except that it will probably be easier to target from existing GPU API's.

Possibly, depending on how low the power actually is. We can't really tell from NVIDIA's tensor cores, because waking up an NVIDIA discrete GPU at all has a higher power cost than running an NPU. Intel's iGPUs have matrix units, but I'm not sure if they can match their NPU on power or performance.

Re: Run LLMs on Apple Neural Engine (ANE)

#84

Post by zozbot234 »

neuroelectron wrote: post #80
ionwake wrote: post #68

Earlier quoted context omitted.

is everyone just waiting for teh DGX Spark? Are they really going to ban local inference?

What do you mean ban? The bandwidth between macs isn't enough to do inference effectively.

I'm pretty sure you can network Macs together via the latest Thunderbolt standards and get pretty decent performance overall. Sure, it will be a bottleneck to some extent but it's still useful for many purposes.

Re: Run LLMs on Apple Neural Engine (ANE)

#85

Post by briandear »

thot_experiment wrote: post #40
jorvi wrote: post #34

Earlier quoted context omitted.

.. who is running LLMs on CPU instead of GPU or TPU/NPU

[flagged]

Can we stop with the derisive “fanboy” nonsense? Most people don’t say “FOSS” fanboy or Linux “fanboy” — but plenty of people here are exactly that. It’s a bit insulting to people that like and appreciate Mac hardware; just because you might not like it doesn’t mean you have to be so dismissive. And that Mac Studio is a very impressive computer — but it’s usually the ones that have never used on that seem to have to most opinions about them.

Re: Run LLMs on Apple Neural Engine (ANE)

#86

Post by wmf »

cowmix wrote: post #14

This sorta reminds me of the lie that was pushed when the Snapdragon X laptops were being released last year. Qualcomm implied the NPU would be used for LLMs — and I bought into the BS without looking into it. I still use a Snapdragon laptop as my daily driver (it's fine) but for running models locally, it's still a joke. Despite Qualcomm's claims about running 13B parameter models, software like LM Studio only runs…

AFAIK Windows 11 does use the NPU to run Phi Silica language models and this is available to any app through some API. The models are quite small as you said though.

Re: Run LLMs on Apple Neural Engine (ANE)

#87

Post by wmf »

rz2k wrote: post #66
fmajid wrote: post #39

Earlier quoted context omitted.

Yes, but their refusal to open up the ANE to third-party models negates that. You can get (or will be able to very soon) a Strix Halo Ryzen AI Max+ 395 able to access 96GB of unified RAM (on a 128GB system) for well under half what you'd pay for an equivalent M4 system from Apple.

Doesn't the M3 Ultra have 3-4x the RAM bandwidth though?

At 3-4x the price.

Re: Run LLMs on Apple Neural Engine (ANE)

#88

Post by wmf »

fmajid wrote: post #39
daedrdev wrote: post #26

Apple is a competitive choice simply because their unified memory allows you to get enough ram that would take multiple Gpus to have enough space to run larger models.

Yes, but their refusal to open up the ANE to third-party models negates that. You can get (or will be able to very soon) a Strix Halo Ryzen AI Max+ 395 able to access 96GB of unified RAM (on a 128GB system) for well under half what you'd pay for an equivalent M4 system from Apple.

Hobbyists don't use ANE on Apple Silicon and they won't use XDNA on Strix Halo either. In both cases the GPU is faster and can access most of system memory.

$2,000 vs. $3,500 isn't well under half either.

Re: Run LLMs on Apple Neural Engine (ANE)

#90

Post by anon373839 »

ljosifov wrote: post #49
behnamoh wrote: post #6

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…

This more than anything feels emblematic to me, that Apple executives are brain dead when it comes to software. AI seemingly being a step too far. (in the s/w direction.) While they could at some level grok classical s/w, NN-s are that Terra Incognita where Apple executives can not possibly follow in. It's just too strange and mysterious world for them to be able to effectively decide or execute on anything. Worked (…

Huh? Apple ships heaps of deep learning-based products/features, and they’ve done so for years. But I’ll agree with you, they’re currently behind in generative AI.
Post reply on HN