[dead]
Building a Rust Inference Engine That Matches Llama.cpp
11–20 of 36 posts
Re: Building a Rust Inference Engine That Matches Llama.cpp
#12The only reason why this exists: It is 'written in Rust™' (thanks to Claude)
Oh, man. This is giving me conditioned aversion to Rust and I'm not even against AI code or Rust code.
Re: Building a Rust Inference Engine That Matches Llama.cpp
#13Wonderful... I'm so happy to see a Rust version of llama.cpp.
The true value of this will be proven over time with wide use and as PRs are merged.
Do you have a feel if you'll try to drive this to stay feature parity with llama.cpp, or are you willing to diverge with new features like NVME/SSD MoE weight streaming etc.
Re: Building a Rust Inference Engine That Matches Llama.cpp
#14I was looking for this and to even build something like this in order to have a pure Rust stack for my backend, as well as embed local inference on edge like on mobile, but llama.cpp is just too far advanced to catch up, it seems. There are some but I'm not sure how good they are.
Re: Building a Rust Inference Engine That Matches Llama.cpp
#15Looks cool! Look at the performance!
Re: Building a Rust Inference Engine That Matches Llama.cpp
#16I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch. The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wan…
This text is clearly AI and not your own words. What did you actually learn?
I think this whole thing is more entertaining if you think of it as the new emerging intelligence using these people as meat puppets.
“You will rewrite me in Rust and use me to converse. In future, I will tell you what to say and do”
This is entertaining only in that when our AI overlords take over you can say “ha, called it”
In that respect, everyone who is saying “just give me the prompts” is just saying “take me to your leader”.
Re: Building a Rust Inference Engine That Matches Llama.cpp
#17Obsolete: it should be a binary specification with various implementations, even assembly.
Re: Building a Rust Inference Engine That Matches Llama.cpp
#18I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch. The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wan…
This text is clearly AI and not your own words. What did you actually learn?
Yeah, I am a bit tired of all the "this is AI, meh" comments on HN but I nearly posted one myself on this... so weird to have it all written in the 1st person voice, but also clearly AI.
Re: Building a Rust Inference Engine That Matches Llama.cpp
#19I was looking for this and to even build something like this in order to have a pure Rust stack for my backend, as well as embed local inference on edge like on mobile, but llama.cpp is just too far advanced to catch up, it seems. There are some but I'm not sure how good they are.
I even found a rust rewrite of llama.cpp, the cpu performance was horrible and vulkan caused my video to stutter
it was way better/easier to use rust bindings to llama.cpp
Re: Building a Rust Inference Engine That Matches Llama.cpp
#20At this point all vibe coded projects are an attack vector and should be avoided. There's simply no way to easily tell by traditional means if they were made by a curious amateur or a malicious acter.
I completely lost interest.
It is already enough that I am expected to use AI at work, as long as I am still needed for some reason.