Building a Rust Inference Engine That Matches Llama.cpp
fratepietro.com
Building a Rust Inference Engine That Matches Llama.cpp
1–10 of 36 posts
Re: Building a Rust Inference Engine That Matches Llama.cpp
#2I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch.
The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wanted to understand inference at a level deeper than “run the binary,” and I wanted a project where every performance claim had to be earned against a real, well-known baseline rather than asserted.
Re: Building a Rust Inference Engine That Matches Llama.cpp
#3I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch. The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wan…
This text is clearly AI and not your own words. What did you actually learn?
Re: Building a Rust Inference Engine That Matches Llama.cpp
#4I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch. The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wan…
This text is clearly AI and not your own words. What did you actually learn?
The M dash stands out, as does "no bindings, ..." but I feel like the LLM would have gotten the correct use of the question mark when they rhetorically ask "why?"
Re: Building a Rust Inference Engine That Matches Llama.cpp
#5I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch. The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wan…
This text is clearly AI and not your own words. What did you actually learn?
hopefully it was produced by their own inference engine!
Re: Building a Rust Inference Engine That Matches Llama.cpp
#6The entire internet is becoming slop and it’s only getting worse
Re: Building a Rust Inference Engine That Matches Llama.cpp
#7The entire internet is becoming slop and it’s only getting worse
Re: Building a Rust Inference Engine That Matches Llama.cpp
#8The only reason why this exists:
It is 'written in Rust™' (thanks to Claude)Re: Building a Rust Inference Engine That Matches Llama.cpp
#9I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch. The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wan…
This text is clearly AI and not your own words. What did you actually learn?
[deleted]
Re: Building a Rust Inference Engine That Matches Llama.cpp
#10At this point all vibe coded projects are an attack vector and should be avoided.
There's simply no way to easily tell by traditional means if they were made by a curious amateur or a malicious acter.