111–120 of 132 posts
Earlier quoted context omitted.
I'm pretty sure you can network Macs together via the latest Thunderbolt standards and get pretty decent performance overall. Sure, it will be a bottleneck to some extent but it's still useful for many purposes.
Yes you can do that and shard a very large model across the devices but it's way too slow so you will get no performance gains beyond being able to run a much larger model at all.
Earlier quoted context omitted.
People keep saying this but I'm not seeing the big difference with other NPU varieties. Either way we're still talking about very experimental stuff that also tends to be hardwired towards some pre-determined use case. So I'm not surprised that people are running into problems while trying to make these more broadly useful.
True; everybody's NPU hardware is afflicted by awkward hardware and software constraints that don't come close to keeping pace with the rapidly-shifting interests of ML researchers. To some degree, that's an unavoidable consequence of how long it takes to design and ship specialized hardware with a supporting software stack. By contrast, ML research is moving way faster because they hardly ever ship anything product-…
Earlier quoted context omitted.
Whisper.cpp has a coreml option which gives 3x speed up over cpu only according to the docs: https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#c...
.. who is running LLMs on CPU instead of GPU or TPU/NPU
Earlier quoted context omitted.
Onnxruntime supports CoreML, though if my experience with converting an embedding model to CoreML using Apple's CoreML conversion tool is similar to the ORT maintainers', I can see why it would be unmaintained. It took multiple tries to get the model to convert at all to the mlpackage format, and then a lot of experimenting to get it to run on the ANE instead of the GPU, only to discover that constant reshaping was k…
ONNX is horrible for anything that has variable input shapes and that is why nobody uses it for LLMs. It fundamentally is poorly designed for anything that doesn't take a fixed size image.
Also the ANE models are limited to 512 tokens of context, so unlikely yet to use these in production.
The README lacks the most important thing: how many more tokens/sec at the same quantization, compared to llama.cpp / MLX? It is worth to switch default platforms only if there is a major improvement.
I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given it…
Not a public follow-up but the iOS 17 speech-to-text model has a clever approach to KV caching that works within the ANE’s constraints (fixed size inputs). I wrote about it here[0] but the gist is you can have a fixed size cache and slide it in chunks with each inference. Not as efficient as a cache that grows by one each time of course. [0]: https://stephenpanaro.com/blog/inside-apples-2023-transforme...
I'm trying to figure out what the secret sauce for this is. It depends on https://github.com/apple/coremltools - is that the key trick or are there other important techniques going on here?
coremltools is the only way to run on ANE, so less of a trick and more of a requirement. The tricks are more around optimizing for the hardware capabilities/constraints. For instance: - conv2d is faster than linear (see Apple's post [0]) so you rewrite the model for that (example from the repo [1]) - inputs/outputs are static shapes, so KV cache requires some creativity (I wrote about that here [2]) - compute is floa…
Earlier quoted context omitted.
coremltools is the only way to run on ANE, so less of a trick and more of a requirement. The tricks are more around optimizing for the hardware capabilities/constraints. For instance: - conv2d is faster than linear (see Apple's post [0]) so you rewrite the model for that (example from the repo [1]) - inputs/outputs are static shapes, so KV cache requires some creativity (I wrote about that here [2]) - compute is floa…
Sounds like M2-era onward have bfloat16: https://eclecticlight.co/2024/01/13/how-m1-macs-may-lag-behi...
Earlier quoted context omitted.
I was referring to both the lower memory bandwidth and lower FLOPs. The GPU can just do… more at once? For now. Or is that changing? I had also assumed that loading a chunk from the cache was not free because I’ve seen cache eviction on my M1, but it’s good to know that it’s no longer as big of a limitation. also, I’m a big fan of your work! I played around with your ModernBERT CoreML port a bit ago
For single batch inference of anything remotely LLM you'll hit the memory bound way before FLOPs, so I haven't actually looked at FLOPs much. For raw performance GPU is certainly better. ANE is more energy efficient, but you need larger batches to really benefit. Maybe cache is the wrong word. This is a limit to how much can be mmap'd for the ANE at once. It's not too hard to hit on M1 if your model is in the GB rang…