"trymirai", every time I hear the word Mirai I think of the large IOT DDoS botnet. Maybe it's just me though.
Show HN: We made our own inference engine for Apple Silicon
11–20 of 48 posts
Re: Show HN: We made our own inference engine for Apple Silicon
#12I'm curious about why the performance gains mentioned were so substantial for Qwen vs Llama?
Re: Show HN: We made our own inference engine for Apple Silicon
#13Amazing! How was your experience using Rust on this project? I'm considering a project in an adjacent space and I'm trying to decide between Rust, C, and Zig. Rust seems a bit burdensome with its complexity compared to C and Zig. Reminds me of C++ in its complexity (although not as bad). I find it difficult to walk through and understand a complicated Rust repository. I don't have that problem with C and Zig for the…
You are confusing familiarity with intrinsic complexity. I have 20 years experience with C/C++ before switching to rust a few years ago. After the initial hurdle, it is way easier and very simple to follow.
Re: Show HN: We made our own inference engine for Apple Silicon
#14Wondering why use Rust other than C++
Why use C++?
Re: Show HN: We made our own inference engine for Apple Silicon
#15Can you explain the type of quantization you support?
would https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally be faster with mirai?
Re: Show HN: We made our own inference engine for Apple Silicon
#16In practice, how often do the models use the ANE? It sounds like you are optimizing for speed which in my experience always favors GPU.
Re: Show HN: We made our own inference engine for Apple Silicon
#17In practice, how often do the models use the ANE? It sounds like you are optimizing for speed which in my experience always favors GPU.
You're right, modern edge devices are powerful enough to run small models, so the real bottleneck for a forward pass is usually memory bandwidth, which defines the upper theoretical limit for inference speed. Right now, we've figured out how to run computations in a granular way on specific processing units, but we expect the real benefits to come later when we add support for VLMs and advanced speculative decoding, where you process more than one token at a time
Re: Show HN: We made our own inference engine for Apple Silicon
#18Can you explain the type of quantization you support? would https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally be faster with mirai?
right now, we support AWQ but are currently working on various quantization methods in https://github.com/trymirai/lalamo
Re: Show HN: We made our own inference engine for Apple Silicon
#19I'm curious about why the performance gains mentioned were so substantial for Qwen vs Llama?
it looks like llama.cpp has some performance issues with bf16
Re: Show HN: We made our own inference engine for Apple Silicon
#20What are the units on the benchmark results? I’m guessing higher is better?