Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Rust Inference Engine That Matches Llama.cpp

fratepietro.com

Post reply on HN
Print view View original

21–30 of 37 posts

Re: Building a Rust Inference Engine That Matches Llama.cpp

#21

Post by lenkite »

brians wrote: post #3
antonellof wrote: post #2

I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch. The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wan…

This text is clearly AI and not your own words. What did you actually learn?

You are talking to the Borg. It has now acquired one more citizen. Soon, we will all be assimilated.

Re: Building a Rust Inference Engine That Matches Llama.cpp

#22

Post by pjmlp »

sylware wrote: post #17

Obsolete: it should be a binary specification with various implementations, even assembly.

Eventually it will be consistent enough to go straight into Assembly.

Until then languages have lost relevance for the most part, it is a matter to configure the model for the desired output language.

This in workflows that require generating an executable, for microservices orchestration, it suffices no code graphical connections.

Re: Building a Rust Inference Engine That Matches Llama.cpp

#23

Post by whythismatters »

anentropic wrote: post #18
brians wrote: post #3

Earlier quoted context omitted.

This text is clearly AI and not your own words. What did you actually learn?

Yeah, I am a bit tired of all the "this is AI, meh" comments on HN but I nearly posted one myself on this... so weird to have it all written in the 1st person voice, but also clearly AI.

Dunno, I think it's still useful to expose slop, chances are that the actual product is low quality/effort as well (i.e I will not be wasting my time checking this project). On a personal note, I've gotten allergic to the word "honest", it's starting to feel like a large scale social engineering psy-op by the Honest Guys™.

Re: Building a Rust Inference Engine That Matches Llama.cpp

#24

Post by j45 »

pjmlp wrote: post #20
noman-land wrote: post #10

At this point all vibe coded projects are an attack vector and should be avoided. There's simply no way to easily tell by traditional means if they were made by a curious amateur or a malicious acter.

I completely lost interest. It is already enough that I am expected to use AI at work, as long as I am still needed for some reason.

It's important to not assume AI can do everything humans can, just because people are saying it.

Re: Building a Rust Inference Engine That Matches Llama.cpp

#25

Post by pjmlp »

j45 wrote: post #24
pjmlp wrote: post #20

Earlier quoted context omitted.

I completely lost interest. It is already enough that I am expected to use AI at work, as long as I am still needed for some reason.

It's important to not assume AI can do everything humans can, just because people are saying it.

I have already seen the real impact on delivery team sizes, no need to hear to what people are saying about what AI can do or not.

Translation and asset creation team members, gone.

Amount of FE reduced, with some projects having now a single BE dev, between AI buddy and ready made SaaS products.

Re: Building a Rust Inference Engine That Matches Llama.cpp

#27

Post by [deleted] »

arjie wrote: post #16
brians wrote: post #3

Earlier quoted context omitted.

This text is clearly AI and not your own words. What did you actually learn?

I think this whole thing is more entertaining if you think of it as the new emerging intelligence using these people as meat puppets. “You will rewrite me in Rust and use me to converse. In future, I will tell you what to say and do” This is entertaining only in that when our AI overlords take over you can say “ha, called it” In that respect, everyone who is saying “just give me the prompts” is just saying “take me t…

[deleted]

Re: Building a Rust Inference Engine That Matches Llama.cpp

#28

Post by rahen »

Four day old repo with 92K lines added per day, okay...

There is basically no evidence of human competence in this repo. This is yet another "rewrite in Rust with Claude" project that brings nothing to the table, but devalues expert work by mimicking competence without the expertise.

Re: Building a Rust Inference Engine That Matches Llama.cpp

#29

Post by antonellof »

antonellof wrote: post #2

I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch. The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wan…

I have the skills, qualifications and experience to do this; ( see https://www.credly.com/users/antonello-fratepietro ) by using AI (I monitor what it does), I’m able to carry out a project like this! I’m a coder, not a blogger; it’s only natural that I use AI, just like everyone else, to write technical texts. I have over 20 years’ experience and I’m not ashamed to admit that I use Claude, Cursor and so on. The problem is not using it. It’s not the code or who writes it that matters, but the result: building my own inference engine. ( reply written with my brain ) ^_^

Re: Building a Rust Inference Engine That Matches Llama.cpp

#30

Post by antonellof »

rockwotj wrote: post #5
brians wrote: post #3

Earlier quoted context omitted.

This text is clearly AI and not your own words. What did you actually learn?

hopefully it was produced by their own inference engine!

That’s the goal, an infinite loop of self improving inference engine that gets better and better by himself.
Post reply on HN