71–80 of 132 posts
Earlier quoted context omitted.
AnythingLLM uses NPU
Could you provide a pointer to docs for this? It wasn't obvious from an initial read of their docs.
I'm trying to figure out what the secret sauce for this is. It depends on https://github.com/apple/coremltools - is that the key trick or are there other important techniques going on here?
The tricks are more around optimizing for the hardware capabilities/constraints. For instance:
- conv2d is faster than linear (see Apple's post [0]) so you rewrite the model for that (example from the repo [1])
- inputs/outputs are static shapes, so KV cache requires some creativity (I wrote about that here [2])
- compute is float16 (not bfloat16) so occasionally you have to avoid activation overflows
[0]: https://machinelearning.apple.com/research/neural-engine-tra...
[1]: https://github.com/Anemll/Anemll/blob/4bfa0b08183a437e759798...
[2]: https://stephenpanaro.com/blog/kv-cache-for-neural-engine
Earlier quoted context omitted.
People keep saying this but I'm not seeing the big difference with other NPU varieties. Either way we're still talking about very experimental stuff that also tends to be hardwired towards some pre-determined use case. So I'm not surprised that people are running into problems while trying to make these more broadly useful.
True; everybody's NPU hardware is afflicted by awkward hardware and software constraints that don't come close to keeping pace with the rapidly-shifting interests of ML researchers. To some degree, that's an unavoidable consequence of how long it takes to design and ship specialized hardware with a supporting software stack. By contrast, ML research is moving way faster because they hardly ever ship anything product-…
GPU's are gaining their own kinds of specialized blocks such as matrix/tensor compute units, or BVH acceleration for ray-tracing (that may or may not turn out to be useful for other stuff). So I'm not sure that there's any real distinction from that POV - a specialized low-power unit in an iGPU is going to be practically indistinguishable from a NPU, except that it will probably be easier to target from existing GPU API's.
Earlier quoted context omitted.
> LLM performance is twice as fast as RTX 5090 your tests are wrong. you used MLX for Mac Studio (optimized for Apple Silicon) but you didn't use vLLM for 5090. There's no way a machine with half the bandwidth of 5090 delivers twice as fast tok/s.
Unless it’s a large model that doesn’t fit in the 5090, bust that’s no longer a $4k macstudio I think.
That's 3x the RAM of the 5090.
Earlier quoted context omitted.
And that still performs worse than entry-level Nvidia gaming cards. Apple isn't serious about AI and needs to figure their AI story out. Every other big tech company is doing something about it.
Not for inferencing. M3 Ultra runs big LLMs twice as fast as RTX 5090. https://creativestrategies.com/mac-studio-m3-ultra-ai-workst... RTX 5090 only has 32GB RAM. M3 Ultra has up to 512 GB with 819 GB/sec bandwidth. It can run models that will not fit on an RTX card. EDIT: Benchmark may not be properly utilizing the 5090. But the M3 Ultra is way more capable than an entry level RTX card at LLM inferencing.
Nvidia makes an incredible product, but apples different market segmentation strategy might make it a real player in the long run.
I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.
At least one link/benchmark I saw said the ANE can be 7x faster than GPU (Metal / MPS), https://discuss.pytorch.org/t/apple-neural-engine-ane-instea... It seems intuitive that if they design hardware very specifically for these applications (beyond just fast matmuls on a GPU), they could squeeze out more performance.
It's about performance/power ratios.
I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.
Eyeballing 3rd party annotated die shots [1], it’s about the size of two GPU cores, but achieves 15.8 tflops. Which is more than the reported 14.7 tflops of the 32-core GPU in the binned M4 Max. [1] https://vengineer.hatenablog.com/entry/2024/10/13/080000
I always felt that the neural engine was wasted silicon, they could add more gpu cores in that die space and redirect the neural processing api to the gpu as needed. But I'm no expert, so if anyone here has a different opinion I'd love to learn from it.
If you did that, you'd stumble into the Apple GPU's lack of tensor acceleration hardware. For an Nvidia-like experience you'd have to re-architecture the GPU to subsume the NPU's role, and if that was easy then everyone would have done it by now.
btw, don't bother trying to buy a bunch of Mac boxes to run LLMs in parallel because it won't be any faster than a single box.
is everyone just waiting for teh DGX Spark? Are they really going to ban local inference?