101–110 of 132 posts
Is there a performance benefit for inference speed on M-series MacBooks, or is the primary task here simply to get inference working on other platforms (like iOS)? If there is a performance benefit, it would be great to see tokens/s of this vs. Ollama.
mlx is much faster, but anemll appeared to use only 500MB of memory compared to the 8GB mlx used.
Even 1gb model is prohibitively big for phones if you want mass adoption.
See my other comments. anemll appears to use less memory.
[0] https://huggingface.co/anemll/anemll-llama-3.2-1B-iOSv2.0
Earlier quoted context omitted.
What do you mean by less wide? The main bottleneck for transformers is memory bandwidth. ANE has a much lower ceiling than CPU/GPU (yes, despite unified memory). Chunking is actually beneficial as long as all the chunks can fit into the ANE’s cache. It speeds up compilation for large network graphs and cached loads are negligible cost. On M1 the cache limit is 3-4GB, but it is higher on M2+.
I was referring to both the lower memory bandwidth and lower FLOPs. The GPU can just do… more at once? For now. Or is that changing? I had also assumed that loading a chunk from the cache was not free because I’ve seen cache eviction on my M1, but it’s good to know that it’s no longer as big of a limitation. also, I’m a big fan of your work! I played around with your ModernBERT CoreML port a bit ago
Maybe cache is the wrong word. This is a limit to how much can be mmap'd for the ANE at once. It's not too hard to hit on M1 if your model is in the GB range. Chunking the model into smaller pieces makes it more likely to "fit", but if it doesn't fit you have to unmap/remap in each forward pass which will be noticeable.
Awesome to hear about ModernBERT! Big fan of your work as well :)
The README lacks the most important thing: how many more tokens/sec at the same quantization, compared to llama.cpp / MLX? It is worth to switch default platforms only if there is a major improvement.
I ran R1-8B for both anemll[0] and mlx[1][2] models on an M4 Max. Prompt: "Tell me a long story about the origins of 42 being the answer." anemll: 9.3 tok/sec, ~500MB of memory used. mlx 8bit: 31.33 tok/sec, ~8.5GB of memory used. mlx bf16: 27.17 tok/sec, ~15.7GB of memory used. Memory results are from activity monitor across any potentially involved processes, but I feel like I might missing something here... [0] ht…
But coreml utilizes ANE, right? Is there some bottleneck in coreml that requires lower level access?
Earlier quoted context omitted.
I ran R1-8B for both anemll[0] and mlx[1][2] models on an M4 Max. Prompt: "Tell me a long story about the origins of 42 being the answer." anemll: 9.3 tok/sec, ~500MB of memory used. mlx 8bit: 31.33 tok/sec, ~8.5GB of memory used. mlx bf16: 27.17 tok/sec, ~15.7GB of memory used. Memory results are from activity monitor across any potentially involved processes, but I feel like I might missing something here... [0] ht…
What hardware are you on? Most models are memory bandwidth limited. ANE was limited to 64GB/s prior to M3 Max or M4 pro. If you are on M1, GPU will be significantly faster for 3-8B models due to memory bandwidth rather then ANE capabilities.
The README lacks the most important thing: how many more tokens/sec at the same quantization, compared to llama.cpp / MLX? It is worth to switch default platforms only if there is a major improvement.
I ran R1-8B for both anemll[0] and mlx[1][2] models on an M4 Max. Prompt: "Tell me a long story about the origins of 42 being the answer." anemll: 9.3 tok/sec, ~500MB of memory used. mlx 8bit: 31.33 tok/sec, ~8.5GB of memory used. mlx bf16: 27.17 tok/sec, ~15.7GB of memory used. Memory results are from activity monitor across any potentially involved processes, but I feel like I might missing something here... [0] ht…
I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…
Onnxruntime supports CoreML, though if my experience with converting an embedding model to CoreML using Apple's CoreML conversion tool is similar to the ORT maintainers', I can see why it would be unmaintained. It took multiple tries to get the model to convert at all to the mlpackage format, and then a lot of experimenting to get it to run on the ANE instead of the GPU, only to discover that constant reshaping was k…