Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run LLMs on Apple Neural Engine (ANE)

github.com

Post reply on HN
Print view View original

121–130 of 132 posts

Re: Run LLMs on Apple Neural Engine (ANE)

#121

Post by jsjohnst »

neuroelectron wrote: post #80
ionwake wrote: post #68

Earlier quoted context omitted.

is everyone just waiting for teh DGX Spark? Are they really going to ban local inference?

What do you mean ban? The bandwidth between macs isn't enough to do inference effectively.

> The bandwidth between macs isn't enough to do inference effectively.

While it’s certainly no where near the memory bandwidth, 80Gbps is on par with most high end, but still affordable, machine to machine connections. Then add on the fact you can have hundreds of gigabytes of shared ram on each machine.

Re: Run LLMs on Apple Neural Engine (ANE)

#122

Post by eyeris »

cowmix wrote: post #14

This sorta reminds me of the lie that was pushed when the Snapdragon X laptops were being released last year. Qualcomm implied the NPU would be used for LLMs — and I bought into the BS without looking into it. I still use a Snapdragon laptop as my daily driver (it's fine) but for running models locally, it's still a joke. Despite Qualcomm's claims about running 13B parameter models, software like LM Studio only runs…

Agreed. Same with intel’s NPUs. I’ve been testing with my intel core evo 155x. The npu only runs int8 as well. At least, Intel has put in a decent amount of effort into the ecosystem

There are a couple ways to interface — DirectML by MS and Intel’s native api (they provide OpenVINO model conversion to convert normal Python ml models) I’ve tried ONNXRuntime conversions for both backends to little success. Additionally the OpenVINO model conversion seems to break the model if the model small enough.

OpenVINO model server seems pretty polished and has openapi compatible endpoints.

Re: Run LLMs on Apple Neural Engine (ANE)

#123

Post by neuroelectron »

ionwake wrote: post #111
neuroelectron wrote: post #95

Earlier quoted context omitted.

Yes you can do that and shard a very large model across the devices but it's way too slow so you will get no performance gains beyond being able to run a much larger model at all.

thats a performance gain

it's paying more for less performance.

Re: Run LLMs on Apple Neural Engine (ANE)

#126

Post by srigi »

SparkyMcUnicorn wrote: post #100
antirez wrote: post #28

The README lacks the most important thing: how many more tokens/sec at the same quantization, compared to llama.cpp / MLX? It is worth to switch default platforms only if there is a major improvement.

I ran R1-8B for both anemll[0] and mlx[1][2] models on an M4 Max. Prompt: "Tell me a long story about the origins of 42 being the answer." anemll: 9.3 tok/sec, ~500MB of memory used. mlx 8bit: 31.33 tok/sec, ~8.5GB of memory used. mlx bf16: 27.17 tok/sec, ~15.7GB of memory used. Memory results are from activity monitor across any potentially involved processes, but I feel like I might missing something here... [0] ht…

Can you add a recent build of llama.cpp (arm64) to the results pool? I'm really interested in comparing mlx to llama.cpp, but setting up the mlx seems too difficult for me to do by myself.

Re: Run LLMs on Apple Neural Engine (ANE)

#127

Post by ericboehs »

SparkyMcUnicorn wrote: post #100
antirez wrote: post #28

The README lacks the most important thing: how many more tokens/sec at the same quantization, compared to llama.cpp / MLX? It is worth to switch default platforms only if there is a major improvement.

I ran R1-8B for both anemll[0] and mlx[1][2] models on an M4 Max. Prompt: "Tell me a long story about the origins of 42 being the answer." anemll: 9.3 tok/sec, ~500MB of memory used. mlx 8bit: 31.33 tok/sec, ~8.5GB of memory used. mlx bf16: 27.17 tok/sec, ~15.7GB of memory used. Memory results are from activity monitor across any potentially involved processes, but I feel like I might missing something here... [0] ht…

Interesting. Does this mean larger models could be ran on less memory? It looks like it uses 15-20x less memory. Could a 671B DeepSeek R1 be ran in just ~40-50GB of memory? It sounds like it'd be 1/3 as fast though (<1tk/sec).

Re: Run LLMs on Apple Neural Engine (ANE)

#128

Post by SparkyMcUnicorn »

srigi wrote: post #126
SparkyMcUnicorn wrote: post #100

Earlier quoted context omitted.

I ran R1-8B for both anemll[0] and mlx[1][2] models on an M4 Max. Prompt: "Tell me a long story about the origins of 42 being the answer." anemll: 9.3 tok/sec, ~500MB of memory used. mlx 8bit: 31.33 tok/sec, ~8.5GB of memory used. mlx bf16: 27.17 tok/sec, ~15.7GB of memory used. Memory results are from activity monitor across any potentially involved processes, but I feel like I might missing something here... [0] ht…

Can you add a recent build of llama.cpp (arm64) to the results pool? I'm really interested in comparing mlx to llama.cpp, but setting up the mlx seems too difficult for me to do by myself.

I ran them again several times to make sure the results were fair. My previous runs also had a different 30B model loaded in the background that I forgot about.

LM Studio is an easy way to use both mlx and llama.cpp

anemll [0]: ~9.3 tok/sec

mlx [1]: ~50 tok/sec

gguf (llama.cpp b5219) [2]: ~41 tok/sec

[0] https://huggingface.co/anemll/anemll-DeepSeekR1-8B-ctx1024_0...

[1] https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Lla...

[2] (8bit) https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-...

Re: Run LLMs on Apple Neural Engine (ANE)

#129

Post by srigi »

SparkyMcUnicorn wrote: post #128
srigi wrote: post #126

Earlier quoted context omitted.

Can you add a recent build of llama.cpp (arm64) to the results pool? I'm really interested in comparing mlx to llama.cpp, but setting up the mlx seems too difficult for me to do by myself.

I ran them again several times to make sure the results were fair. My previous runs also had a different 30B model loaded in the background that I forgot about. LM Studio is an easy way to use both mlx and llama.cpp anemll [0]: ~9.3 tok/sec mlx [1]: ~50 tok/sec gguf (llama.cpp b5219) [2]: ~41 tok/sec [0] https://huggingface.co/anemll/anemll-DeepSeekR1-8B-ctx1024_0... [1] https://huggingface.co/mlx-community/DeepSeek-…

Thank you very much.

Re: Run LLMs on Apple Neural Engine (ANE)

#130

Post by ycomby321 »

Something to consider for deploying LLMs on the ANE is: https://github.com/pytorch/executorch/tree/main/examples/app...

The model does have some limitations (e.g., need for QAT for 4-bit quantization), lack of a C++ runner to execute the model, but parts of the model are promising.

If interested in further discussion, join the conversation on the ExecuTorch discord channel: https://discord.gg/xHxqsD5b

Post reply on HN