Show HN: We made our own inference engine for Apple Silicon • Hacker News Board

Re: Show HN: We made our own inference engine for Apple Silicon

#31

Post by giancarlostoro » Tue, Jul 15, 2025, 6:29 PM UTC

cwlcwlcwlingg wrote: post #8

Wondering why use Rust other than C++

...or D? or Go? or Java? C#? Zig? etc they chose what they were most comfortable with. Rust is fine, it's not for everyone clearly, but those who use it produce high quality software, I would argue similar with Go, without all the unnecessary mental overhead of C or C++

Re: Show HN: We made our own inference engine for Apple Silicon

#32

Post by outworlder » Tue, Jul 15, 2025, 6:42 PM UTC

cwlcwlcwlingg wrote: post #8

Wondering why use Rust other than C++

Why use C++ for greenfield projects?

Re: Show HN: We made our own inference engine for Apple Silicon

#33

Post by fnord77 » Tue, Jul 15, 2025, 6:51 PM UTC

greggh wrote: post #11

"trymirai", every time I hear the word Mirai I think of the large IOT DDoS botnet. Maybe it's just me though.

I think of the goofy Toyota fuel cell car. I think a grand total of about 6 have been sold (leased) in california

Re: Show HN: We made our own inference engine for Apple Silicon

#34

Post by nodesocket » Tue, Jul 15, 2025, 7:16 PM UTC

I just spun up a AWS EC2 g6.xlarge instance to do some llm work. The GPU is NVIDIA L4 24GB and costs $0.8048/per hour. Starting to think about switching to an Apple mac2-m2.metal instance for $0.878/ per hour. Big question is the Mac instance only has 24GB of unified memory.

Re: Show HN: We made our own inference engine for Apple Silicon

#35

Post by floam » Tue, Jul 15, 2025, 7:45 PM UTC

How does this compare to https://github.com/Anemll/Anemll?

Re: Show HN: We made our own inference engine for Apple Silicon

#36

Post by darkolorin » Tue, Jul 15, 2025, 8:54 PM UTC

giancarlostoro wrote: post #30

Hoping the author can answer, I'm still learning about how this all works. My understanding is that inference is "using the model" so to speak. How is this faster than established inference engines specifically on Mac? Are models generic enough that if you build e.g. an inference engine focused on AMD GPUs or even Intel GPUs, would they achieve reasonable performance? I always assumed because Nvidia is king of AI tha…

Basically “faster” means better performance e.g. tokens/s without loosing quality (benchmarks scores for models). So when we say faster we provide more tokens per second than llama cpp. That means we effectively utilize hardware API available (for example we wrote our own kernels) to perform better.

Re: Show HN: We made our own inference engine for Apple Silicon

#37

Post by zackangelo » Tue, Jul 15, 2025, 9:07 PM UTC

We also wrote our inference engine in rust for mixlayer, happy to answer any questions from those trying to do the same.

Looks like this uses ndarray and mpsgraph (which I did not know about!), we opted to use candle instead.

Re: Show HN: We made our own inference engine for Apple Silicon

#38

Post by khurs » Tue, Jul 15, 2025, 10:27 PM UTC

cwlcwlcwlingg wrote: post #8

Wondering why use Rust other than C++

The recommendation from the security agencies is to prefer Rust over C++ as less risk of exploits.

Checked and Lama.cpp used C++ (obviously) and Llama uses Go.

Re: Show HN: We made our own inference engine for Apple Silicon

#39

Post by khurs » Tue, Jul 15, 2025, 10:28 PM UTC

ewuhic wrote: post #6

>faster than llama cpp in all of the use cases What's your deliberate, well-thought roadmap for achieving adoption similar to llama cpp?

Ollama is the leader isn't it?

Brew stats (downloads last 30 days)

Ollama - 28,232 Lama.cpp - 7,826

Re: Show HN: We made our own inference engine for Apple Silicon

#40

Post by khurs » Tue, Jul 15, 2025, 10:29 PM UTC

Have you added it to HomeBrew and other package managers yet?

Also any app deployed to PROD but developed on Mac need to be consistent i.e. work on Linux/in container.