Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we write our own C and C++ inference engines

localai.io

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Why we write our own C and C++ inference engines

#31

Post by aabdi »

I don’t think it would be surprising that people want to write their own kernels.

A big problem with the existing engines like llama or sd is that they don’t support optimal graph compilation. Usually this means about a real 2 or 3x multiplier loss relative to optimal. Cuda graphs do okay but they still leave a lot on the floor

It’s usually worth it to optimize in that context if you are willing to peer into the mechanics.

Of course that’s expensive. You need to know how to appropriately pipeline and merge your kernels.

Re: Why we write our own C and C++ inference engines

#32

Post by pjmlp »

nnevatie wrote: post #29
xienze wrote: post #24

Earlier quoted context omitted.

I've had this debate before on HN. The excuses are generally "well it can write better code than most developers, but an LLM can't write better prose than most people" (I strongly disagree with this) and, what I think is at the heart of the matter, "text is for the reader to read directly, code is hidden." Or in other words, "as long as I can't tell it's AI, it's fine."

I would rather read faulty English, succinct sentences and getting to the point, than the generic filler LLMs produce.

I would also code review code that people actually put some effort learning on how to write it, even if it had one bug or two.

Re: Why we write our own C and C++ inference engines

#33

Post by tovlier »

altmanaltman wrote: post #17
nnevatie wrote: post #9

Earlier quoted context omitted.

The post is full of signs. Here's only a couple of examples: > The method, the measurements, and what it costs us. > That is the general shape of these wins. > Parity is the gate, speed is the follow-up I could go on and on, but you probably get the point. If you don't find anything funny with the above, you might have not been enough-exposed to slop.

What do you mean you could go on and on? Why do you think those sentences are AI written. And okay, your second argument is that I just don't know slop because I am not exposed to it? But you don't know anything about me or what I am exposed. You're just making random claims and stating they are correct without any evidence or arguments.

[dead]

Re: Why we write our own C and C++ inference engines

#34

Post by pjmlp »

nnevatie wrote: post #4
stephbook wrote: post #3

Should have started with writing your own blog posts.

Came here to say the same. Really tiring to read these slop-infested posts, where everything has the “right shape”.

While I am tired to see slop-infested pull requests being celebrated.

Re: Why we write our own C and C++ inference engines

#35

Post by sylware »

lelanthran wrote: post #12
stephbook wrote: post #3

Should have started with writing your own blog posts.

> Should have started with writing your own blog posts. While the page looks vibe-coded[1], the content itself does not have any AI tells. What are the tells you are seeing? [1] Too many sites I find on HN frontpage these days slow my PC to a crawl. I assume they are all using the same autogenerated HTML, Javascrip and CSS to make animated backgrounds :-( On this specific site scrolling is laggy.

Use lynx, links or netsurf. Or vibe code your noscript/basic HTML web engine?

Re: Why we write our own C and C++ inference engines

#36

Post by cataphract »

lelanthran wrote: post #12
stephbook wrote: post #3

Should have started with writing your own blog posts.

> Should have started with writing your own blog posts. While the page looks vibe-coded[1], the content itself does not have any AI tells. What are the tells you are seeing? [1] Too many sites I find on HN frontpage these days slow my PC to a crawl. I assume they are all using the same autogenerated HTML, Javascrip and CSS to make animated backgrounds :-( On this specific site scrolling is laggy.

Just from the subtitle, you can already tell it's likely AI-generated. Maybe you're lucky enough not to be exposed to claude/codex output your whole working day.

Re: Why we write our own C and C++ inference engines

#38

Post by interpol_p »

PatronBernard wrote: post #30
interpol_p wrote: post #20

Earlier quoted context omitted.

I stopped reading almost immediately. The stylistic choices in the writing just felt like LLM to me. Examples: "depth estimation that beats PyTorch on CPU in half the memory" — "…beats X in Y…" "Most LocalAI backends wrap somebody else’s engine, and that is the right default." — "…and that is the right" "MLX and the rest are maintained by people who are better at those models than we are" — "better at those models th…

Why are you using an em-dash though?

I've always used em-dash. I got familiar with the hotkeys when I started using Mac OS X in 2002. Option+Hyphen for en-dash, Option+Shift+Hyphen for em-dash. It's unfortunate that it's an LLM tell. I'm glad the LLMs haven't subsumed proper ellipses yet…

Re: Why we write our own C and C++ inference engines

#39

Post by dreeseaw »

altmanaltman wrote: post #7
stephbook wrote: post #3

Should have started with writing your own blog posts.

I went through the post because of your comment but it really doesn't look like AI slop. Can you please share why you feel like its slop and not written by a human? I can also say "should have started writing your own comments" to you and its unfalsifiable. Blanket accusations with no proof is not a good move really.

"This post is about what those ports buy, measured, and what they cost."

if you've done any amount of optimizaation or hill-climbing work (kernel optimization, autoresearch type shit), you would know that these models LOVE the terms "buy" and "cost" in this sense. they use them nearly constantly (along with "budget", and even "credit" (even when working with RL/credit assignment!))

Re: Why we write our own C and C++ inference engines

#40

Post by sasaf5 »

interpol_p wrote: post #20
lelanthran wrote: post #12

Earlier quoted context omitted.

> Should have started with writing your own blog posts. While the page looks vibe-coded[1], the content itself does not have any AI tells. What are the tells you are seeing? [1] Too many sites I find on HN frontpage these days slow my PC to a crawl. I assume they are all using the same autogenerated HTML, Javascrip and CSS to make animated backgrounds :-( On this specific site scrolling is laggy.

I stopped reading almost immediately. The stylistic choices in the writing just felt like LLM to me. Examples: "depth estimation that beats PyTorch on CPU in half the memory" — "…beats X in Y…" "Most LocalAI backends wrap somebody else’s engine, and that is the right default." — "…and that is the right" "MLX and the rest are maintained by people who are better at those models than we are" — "better at those models th…

"Parity is what makes the replacement a drop-in rather than a migration."

It always reads like a 14 year old arguing.

Post reply on HN