Building a Rust Inference Engine That Matches Llama.cpp • Hacker News Board

Re: Building a Rust Inference Engine That Matches Llama.cpp

#41

Post by Alien1Being » Mon, Aug 10, 2026, 5:00 AM UTC

antonellof wrote: post #35

ubedan wrote: post #13

Wonderful... I'm so happy to see a Rust version of llama.cpp. The true value of this will be proven over time with wide use and as PRs are merged. Do you have a feel if you'll try to drive this to stay feature parity with llama.cpp, or are you willing to diverge with new features like NVME/SSD MoE weight streaming etc.

Thanks for the comment. Parity with llama.cpp isn't my goal. What I actually want is MoE on machines that can't fit the model in VRAM, and specifically expert-level residency instead of layer offload: track which experts get hit during decode, keep those resident, evict the rest. Doing that well needs the router, the KV cache and the memory manager to be designed together, which is about the only good reason to write…

Since you didn't contribute yourself, that is quite funny.

Posting a link to AI slop about a mess of AI vibecode is not contributing.

Re: Building a Rust Inference Engine That Matches Llama.cpp

#42

Post by antonellof » Mon, Aug 10, 2026, 10:30 AM UTC

[flagged]

Re: Building a Rust Inference Engine That Matches Llama.cpp

#43

Post by antonellof » Mon, Aug 10, 2026, 10:44 AM UTC

Alien1Being wrote: post #41

antonellof wrote: post #35

Earlier quoted context omitted.

Thanks for the comment. Parity with llama.cpp isn't my goal. What I actually want is MoE on machines that can't fit the model in VRAM, and specifically expert-level residency instead of layer offload: track which experts get hit during decode, keep those resident, evict the rest. Doing that well needs the router, the KV cache and the memory manager to be designed together, which is about the only good reason to write…

Since you didn't contribute yourself, that is quite funny. Posting a link to AI slop about a mess of AI vibecode is not contributing.

Contribute = Human ideas, testing, review etc. The monkey part of writing code by hand = obsolete. Do you still write code without and IDE? you remember all the programming language words? everything? you use stackoverflow? it's called evolution, btw: AI full disclosure This software is developed with strong assistance from Cursor, Grok 4.5, GPT 5.6, and Claude Fable 5, with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without llama.cpp and GGML, largely written by hand.

Re: Building a Rust Inference Engine That Matches Llama.cpp

#44

Post by sylware » Mon, Aug 10, 2026, 12:01 PM UTC

pjmlp wrote: post #22

sylware wrote: post #17

Obsolete: it should be a binary specification with various implementations, even assembly.

Eventually it will be consistent enough to go straight into Assembly. Until then languages have lost relevance for the most part, it is a matter to configure the model for the desired output language. This in workflows that require generating an executable, for microservices orchestration, it suffices no code graphical connections.

What is generated is not machine code, but 'human' assemby to input into an assembler (was nasm). It is much much better.

Basically, if all that is really true, coding AIs may be our salvation from those abominations which are compilers.

Re: Building a Rust Inference Engine That Matches Llama.cpp

#45

Post by antonellof » Mon, Aug 10, 2026, 12:45 PM UTC

sylware wrote: post #44

pjmlp wrote: post #22

Earlier quoted context omitted.

Eventually it will be consistent enough to go straight into Assembly. Until then languages have lost relevance for the most part, it is a matter to configure the model for the desired output language. This in workflows that require generating an executable, for microservices orchestration, it suffices no code graphical connections.

What is generated is not machine code, but 'human' assemby to input into an assembler (was nasm). It is much much better. Basically, if all that is really true, coding AIs may be our salvation from those abominations which are compilers.

[flagged]