Viewing profile — anemll

Check it out, you might be able to speed it up using this https://github.com/Anemll/anemll-flash-mlx https://x.com/anemll/status/2038684375425200360

17B includes 10 expert plus one shared. So actual size of the expert is much smaller

Check my repo, I had added some support for GUFF/untloth, Q3,Q5/Q8 https://github.com/Anemll/flash-moe/blob/iOS-App/docs/gguf-h...

Thanks for posting this, that's how I first found out about Dan's experiment! SSD speed doubled in the M5P/M generation, that makes it usable! I think one paper under the radar is …

SSD streaming to compute units is new. M4 max can do 15 t/s with its 15GB/s drives

Yes, SSD speed is critical though. The repo has macOS builds for CLI and Desktop. It's early stages though. M4 Max gets 10-15 TPS on 400B depending on quantization. Compute is an i…

multiple NAND, and apple already used it in Mac Studio. Plus better cooling

both, tbh

Probably 2x speed for Mac Studio this year if they do double NAND ( or quad?)

[flagged]

https://xcancel.com/anemll/status/2035901335984611412

Tensor Parallel test with RDMA last week https://x.com/anemll/status/1996349871260107102 Note fast sync workaround

In macOS 26.2 (Tahoe) beta, Apple introduced a low-latency Thunderbolt 5 RDMA driver, enabling up to 80 Gb/s bidirectional bandwidth for Mac clustering—ideal for distributed ML on …

It’s also supported in Apple Neural Engine https://github.com/Anemll/Anemll

We can ran 2000 or 4000 context with ANE

M4 max should work at 120GB for ANE and 500+ for GPU. So GPU will be 3-4 times faster for anything over 1-3B. ANE is likely as fast for prefill due to higher FLOPs

Right.I was thinking about it, you still need batch refill, however, Apple Core ML tools were failing for attention activations quantization. Long context, pre-fill is still comput…

Yes for GPU, however ANE only supports FP16 plus integers. M4/A17 added accelerated int8 that is twice faster than FP16

Memory bandwidth is the main bottleneck. It got better with M3/M4. ANE is really fast in FLOPS but low in memory bandwidth.

What hardware are you on? Most models are memory bandwidth limited. ANE was limited to 64GB/s prior to M3 Max or M4 pro. If you are on M1, GPU will be significantly faster for 3-8B…