Somewhat faster on small models. Requires new format.
Not sure what the goal is for this project? Not seeing how this presents adequate benefits to get adopted by the community
Show HN: We made our own inference engine for Apple Silicon
21–30 of 48 posts
Re: Show HN: We made our own inference engine for Apple Silicon
#22What are the units on the benchmark results? I’m guessing higher is better?
yeah, tokens per second
Re: Show HN: We made our own inference engine for Apple Silicon
#23How does this bench compared to MLX?
Re: Show HN: We made our own inference engine for Apple Silicon
#24In practice, how often do the models use the ANE? It sounds like you are optimizing for speed which in my experience always favors GPU.
You're right, modern edge devices are powerful enough to run small models, so the real bottleneck for a forward pass is usually memory bandwidth, which defines the upper theoretical limit for inference speed. Right now, we've figured out how to run computations in a granular way on specific processing units, but we expect the real benefits to come later when we add support for VLMs and advanced speculative decoding,…
VLMs = very large models?
Re: Show HN: We made our own inference engine for Apple Silicon
#25Earlier quoted context omitted.
You're right, modern edge devices are powerful enough to run small models, so the real bottleneck for a forward pass is usually memory bandwidth, which defines the upper theoretical limit for inference speed. Right now, we've figured out how to run computations in a granular way on specific processing units, but we expect the real benefits to come later when we add support for VLMs and advanced speculative decoding,…
VLMs = very large models?
Probably vision language models.
Re: Show HN: We made our own inference engine for Apple Silicon
#26Somewhat faster on small models. Requires new format. Not sure what the goal is for this project? Not seeing how this presents adequate benefits to get adopted by the community
Written in Rust is a big one for me.
Re: Show HN: We made our own inference engine for Apple Silicon
#27How does this bench compared to MLX?
I use MLX in lmstudio and it doesn't have whatever issues llama cpp is showing here.
Qwen3-0.6B at 5 t/s doesn't make any sense. Something is clearly wrong for that specific model.
Re: Show HN: We made our own inference engine for Apple Silicon
#28Wondering why use Rust other than C++
I wonder why they didn’t use Fortran.
Re: Show HN: We made our own inference engine for Apple Silicon
#29Somewhat faster on small models. Requires new format. Not sure what the goal is for this project? Not seeing how this presents adequate benefits to get adopted by the community
It's utilizing Apple ANE and probably other optimization tools provided by Apple's framework. Not sure if llama.cpp uses them, but if they're not then the benchmark on GitHub says it all.
Re: Show HN: We made our own inference engine for Apple Silicon
#30Hoping the author can answer, I'm still learning about how this all works. My understanding is that inference is "using the model" so to speak. How is this faster than established inference engines specifically on Mac? Are models generic enough that if you build e.g. an inference engine focused on AMD GPUs or even Intel GPUs, would they achieve reasonable performance? I always assumed because Nvidia is king of AI that you had to suck it up, or is it just that most inference engines being used are married to Nvidia?
I would love to understand how universal these models can become.