Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs on Apple Neural Engine (ANE)

github.com

Post reply on HN
Print view View original

91–100 of 132 posts

Re: Run LLMs on Apple Neural Engine (ANE)

#91

Post by balnazzar »

voidspark wrote: post #61
echelon wrote: post #57

Earlier quoted context omitted.

And that still performs worse than entry-level Nvidia gaming cards. Apple isn't serious about AI and needs to figure their AI story out. Every other big tech company is doing something about it.

Not for inferencing. M3 Ultra runs big LLMs twice as fast as RTX 5090. https://creativestrategies.com/mac-studio-m3-ultra-ai-workst... RTX 5090 only has 32GB RAM. M3 Ultra has up to 512 GB with 819 GB/sec bandwidth. It can run models that will not fit on an RTX card. EDIT: Benchmark may not be properly utilizing the 5090. But the M3 Ultra is way more capable than an entry level RTX card at LLM inferencing.

It can run models that cannot fit on TEN rtx 5090s (yes, it can run DeepSeek V3/R1, quantized at 4 bit, at a honest 18-19 tok/s, and that's a model you cannot fit into 10 5090s..).

Re: Run LLMs on Apple Neural Engine (ANE)

#92

Post by balnazzar »

echelon wrote: post #57
kamranjon wrote: post #43

Earlier quoted context omitted.

Pretty sure they're using the 80 GPU cores available in that case.

And that still performs worse than entry-level Nvidia gaming cards. Apple isn't serious about AI and needs to figure their AI story out. Every other big tech company is doing something about it.

Not true. It performs 20-30% better than a RTX A6000 (I have both). Except it has more than 10 times the VRAM. For a comparison with newer Nvidia cards, benchmarks say it does substantially better than a 5070ti, a bit better than a 4080, and a bit worse than a 5080. But once again, it got 30 times the vram amount of the mentioned cards, which for AI workloads are just expensive toys due to lack of vram indeed.

Re: Run LLMs on Apple Neural Engine (ANE)

#93

Post by voidspark »

balnazzar wrote: post #91
voidspark wrote: post #61

Earlier quoted context omitted.

Not for inferencing. M3 Ultra runs big LLMs twice as fast as RTX 5090. https://creativestrategies.com/mac-studio-m3-ultra-ai-workst... RTX 5090 only has 32GB RAM. M3 Ultra has up to 512 GB with 819 GB/sec bandwidth. It can run models that will not fit on an RTX card. EDIT: Benchmark may not be properly utilizing the 5090. But the M3 Ultra is way more capable than an entry level RTX card at LLM inferencing.

It can run models that cannot fit on TEN rtx 5090s (yes, it can run DeepSeek V3/R1, quantized at 4 bit, at a honest 18-19 tok/s, and that's a model you cannot fit into 10 5090s..).

Right, that's the $9500 Mac Studio with 512GB RAM and 80-core GPU.

16x the RAM of RTX 5090.

There are two versions of the M3 Ultra

28-core CPU, 60-core GPU

32-core CPU, 80-core GPU

Both have a 32-core Neural Engine.

Re: Run LLMs on Apple Neural Engine (ANE)

#94

Post by loh »

antirez wrote: post #28

The README lacks the most important thing: how many more tokens/sec at the same quantization, compared to llama.cpp / MLX? It is worth to switch default platforms only if there is a major improvement.

Yeah, I looked all over for a comparison and couldn't find anything in the repo, on their social media, etc. I saw some other comments here that said it's supposed to be "15.8 fp16 ops compared to 14.7 fp32 ops" but that isn't really enough to go on. Maybe when I have the time I'll install their TestFlight app and do some comparisons myself.

Re: Run LLMs on Apple Neural Engine (ANE)

#95

Post by neuroelectron »

zozbot234 wrote: post #84
neuroelectron wrote: post #80

Earlier quoted context omitted.

What do you mean ban? The bandwidth between macs isn't enough to do inference effectively.

I'm pretty sure you can network Macs together via the latest Thunderbolt standards and get pretty decent performance overall. Sure, it will be a bottleneck to some extent but it's still useful for many purposes.

Yes you can do that and shard a very large model across the devices but it's way too slow so you will get no performance gains beyond being able to run a much larger model at all.

Re: Run LLMs on Apple Neural Engine (ANE)

#96

Post by conradev »

kamranjon wrote: post #56
conradev wrote: post #53

Earlier quoted context omitted.

It essentially never makes sense to run on the CPU and you will only ever see enthusiasts doing it. Yes, hammering the GPU too hard can affect the display server, but no, switching to the CPU is not a good alternative

Not switching to the CPU - switching to the ANE (Neural Cores) - if you read the research papers Apple has released - the example I gave is pretty much how it's being used - small image classification models running on the ANE, alongside a graphics app that needs the GPU to be free.

Oh, yes, I misread! It’s great for that

Re: Run LLMs on Apple Neural Engine (ANE)

#97

Post by [deleted] »

ljosifov wrote: post #49
behnamoh wrote: post #6

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…

This more than anything feels emblematic to me, that Apple executives are brain dead when it comes to software. AI seemingly being a step too far. (in the s/w direction.) While they could at some level grok classical s/w, NN-s are that Terra Incognita where Apple executives can not possibly follow in. It's just too strange and mysterious world for them to be able to effectively decide or execute on anything. Worked (…

[deleted]

Re: Run LLMs on Apple Neural Engine (ANE)

#98

Post by mdp2021 »

voidspark wrote: post #75
seanmcdirmid wrote: post #65

Earlier quoted context omitted.

Unless it’s a large model that doesn’t fit in the 5090, bust that’s no longer a $4k macstudio I think.

$4k will get you a 96 GB Mac Studio with M3 Ultra (819 GB/sec). That's 3x the RAM of the 5090.

> That's 3x the RAM of the 5090

And a bit less than half the bandwidth (saying for completeness).

Re: Run LLMs on Apple Neural Engine (ANE)

#99

Post by conradev »

smpanaro wrote: post #59
conradev wrote: post #48

Earlier quoted context omitted.

My understanding is that model throughput is fundamentally limited at some point by the fact that the ANE is less wide than the GPU. At that point, the ANE loses because you have to split the model into chunks and only one fits at a time.

What do you mean by less wide? The main bottleneck for transformers is memory bandwidth. ANE has a much lower ceiling than CPU/GPU (yes, despite unified memory). Chunking is actually beneficial as long as all the chunks can fit into the ANE’s cache. It speeds up compilation for large network graphs and cached loads are negligible cost. On M1 the cache limit is 3-4GB, but it is higher on M2+.

I was referring to both the lower memory bandwidth and lower FLOPs. The GPU can just do… more at once? For now. Or is that changing?

I had also assumed that loading a chunk from the cache was not free because I’ve seen cache eviction on my M1, but it’s good to know that it’s no longer as big of a limitation.

also, I’m a big fan of your work! I played around with your ModernBERT CoreML port a bit ago

Re: Run LLMs on Apple Neural Engine (ANE)

#100

Post by SparkyMcUnicorn »

antirez wrote: post #28

The README lacks the most important thing: how many more tokens/sec at the same quantization, compared to llama.cpp / MLX? It is worth to switch default platforms only if there is a major improvement.

I ran R1-8B for both anemll[0] and mlx[1][2] models on an M4 Max.

Prompt: "Tell me a long story about the origins of 42 being the answer."

anemll: 9.3 tok/sec, ~500MB of memory used.

mlx 8bit: 31.33 tok/sec, ~8.5GB of memory used.

mlx bf16: 27.17 tok/sec, ~15.7GB of memory used.

Memory results are from activity monitor across any potentially involved processes, but I feel like I might missing something here...

[0] https://huggingface.co/anemll/anemll-DeepSeekR1-8B-ctx1024_0...

[1] https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Lla...

[2] https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Lla...

Post reply on HN