Live data from Hacker News

Why we write our own C and C++ inference engines

localai.io

41–50 of 63 posts

Re: Why we write our own C and C++ inference engines

#41

Should have started with writing your own blog posts.

This witch hunting is getting tiring.

I get the motives but it's tiring. I myself sometimes check my own (unpublished) writing or have friends preview it and the same feedback comes out, it's all hand written.

Re: Why we write our own C and C++ inference engines

#42

Should have started with writing your own blog posts.

> Should have started with writing your own blog posts. While the page looks vibe-coded[1], the content itself does not have any AI tells. What are the tells you are seeing? [1] Too many sites I find on HN frontpage these days slow my PC to a crawl. I assume they are all using the same autogenerated HTML, Javascrip and CSS to make animated backgrounds :-( On this specific site scrolling is laggy.

Opened this page in Firefox and my CPU fan started spinning like crazy. All this for a background that just renders the whole page unreadable.

Stop. Seek help.

Re: Why we write our own C and C++ inference engines

#43

Earlier quoted context omitted.

> Should have started with writing your own blog posts. While the page looks vibe-coded[1], the content itself does not have any AI tells. What are the tells you are seeing? [1] Too many sites I find on HN frontpage these days slow my PC to a crawl. I assume they are all using the same autogenerated HTML, Javascrip and CSS to make animated backgrounds :-( On this specific site scrolling is laggy.

I stopped reading almost immediately. The stylistic choices in the writing just felt like LLM to me. Examples: "depth estimation that beats PyTorch on CPU in half the memory" — "…beats X in Y…" "Most LocalAI backends wrap somebody else’s engine, and that is the right default." — "…and that is the right" "MLX and the rest are maintained by people who are better at those models than we are" — "better at those models th…

In "biometrics that match insightface bit for bit" - the "bit for bit" thing is also something I have encountered more than once.

Re: Why we write our own C and C++ inference engines

#44

Should have started with writing your own blog posts.

This witch hunting is getting tiring. I get the motives but it's tiring. I myself sometimes check my own (unpublished) writing or have friends preview it and the same feedback comes out, it's all hand written.

Yeah, weirdly enough, I was testing gptzero on some of my own writings, it suggested they're 100% written by a human, tried this article, it says 100% written by AI. I would prefer for any blog post to be human sourced as much and as often as possible, but there's no true way to enforce or incentivize people to do this sadly.

People on HN have accused me of sounding like an AI one time or another, English is my second language and sometimes my ADD goofs my writing into something that sounds like gibberish even though I fully understand what I wrote others might not, so yeah.

Re: Why we write our own C and C++ inference engines

#46

Should have started with writing your own blog posts.

its so toxic when people show this kind of confidence in their ability to tell when something is ai or not.

It's so toxic when people prevent others from avoiding wasting their time reading blatant fluff that no one spent their own time writing.

Re: Why we write our own C and C++ inference engines

#47
post #4

Earlier quoted context omitted.

Came here to say the same. Really tiring to read these slop-infested posts, where everything has the “right shape”.

The thing is... although the writing is unmistakably full of LLMisms, I can't fault the `author` for having produced a slop readme. The content earns its keep, it only grates because of the robotic personality. We need another word than "slop" for this. "blland", "llame",... ?

Vapid

Re: Why we write our own C and C++ inference engines

#48

Should have started with writing your own blog posts.

> Should have started with writing your own blog posts. While the page looks vibe-coded[1], the content itself does not have any AI tells. What are the tells you are seeing? [1] Too many sites I find on HN frontpage these days slow my PC to a crawl. I assume they are all using the same autogenerated HTML, Javascrip and CSS to make animated backgrounds :-( On this specific site scrolling is laggy.

This has many of the writing tropes of Claude. I'd bet money that much of it is ai-generated, if not all of it. This paragraph alone feels almost entirely Claude's self-congratulatory style of writing:

> We are ahead at all six points, and five of those six are ties. Our run-to-run noise band is 0.5%, and concurrency 2 through 32 land between 0.7% and 1.7%, so the honest reading is that only the single-stream case (4.5%) is clearly outside noise. Output is token-for-token identical to vLLM at every point on that curve. Peak host memory is 24.88 GiB against 28.18 GiB.

Re: Why we write our own C and C++ inference engines

#49

Should have started with writing your own blog posts.

> Should have started with writing your own blog posts. While the page looks vibe-coded[1], the content itself does not have any AI tells. What are the tells you are seeing? [1] Too many sites I find on HN frontpage these days slow my PC to a crawl. I assume they are all using the same autogenerated HTML, Javascrip and CSS to make animated backgrounds :-( On this specific site scrolling is laggy.

[deleted]

Re: Why we write our own C and C++ inference engines

#50
Ignoring the fact that this is clearly not written by a human, it's untrustworthy and the claims are dubious at best.

1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project.

2. The depth anything comparison isn't apples to apples. Of course a q8_0 quant is faster than f32. It's 4x less data to chew on.

3. This point is silly, it again fights against the argument that writing your own c++ engines are worth it. It's a bug, just fix it:

> The reason it is faster has nothing to do with writing better matmul kernels than PyTorch. Two positional embeddings, the DPT head’s UV embedding and the backbone’s bicubic position embedding, were being recomputed on every forward pass with single-threaded scalar sin, cos and bicubic loops, even though they depend only on the input geometry and are identical every call.

They argue against their own point again just after:

> For a biometric pipeline, matching the reference exactly matters more than being faster than it.

Okay, then don't rewrite it! It's not faster anyway!

4. If the argument is that the venv is large, then rewriting it in C++ doesn't seem like the answer, it seems like a lot of work and maintenance to avoid having to cull unreachable files in your venv. In a past life I maintained a simple denylist for files in node_modules. The low hanging fruit is plentiful and generally very safe.

Post reply on HN