Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we write our own C and C++ inference engines

localai.io

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Why we write our own C and C++ inference engines

#51

Post by cyanregiment »

People are critiquing whether or not AI wrote the article.

Well, it's clearly not 1-shot / low effort.

(can we agree there's a difference between low effort 1-shotting a prompt and those who use LLMs interactively?)

The one reason why I can't take a mega hardline stance against AI for writing: Not everyone speaks English as a first language.

People with awful English speaking and writing skills are using LLMs to contribute content in English. This is a good thing. They used to try their best and suffix every post with "sorry for my english".

The author's name is Ettore Di Giacinto - I put 2 and 2 together on this one.

I give a pass when non-native English speakers are just using it to write better English and keep up with the rest of us. It wasn't super obvious anyway, it wasn't interruptive.

The content is beyond that, and I more than welcome stuff like this here.

Re: Why we write our own C and C++ inference engines

#52

Post by bwfan123 »

bastawhiz wrote: post #50

Ignoring the fact that this is clearly not written by a human, it's untrustworthy and the claims are dubious at best. 1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project. 2. The depth…

Take a look at the code. It is a conglomeration of python, rust, go, cpp etc. I was expecting lean C or C++ but all I see is a kitchen sink of code, the AI generated kind. No problem with that, but it is inconsistent with the messaging. I appreciate the intention behind this, but dont like the execution. My prediction is that this will enter the graveyard of ai generated git repos.

For open-source code to get adoption and support, the theory of its construction needs to be understood by humans because at the end of the day, some human will be responsible for its operation. I would call this the Terrence Tao principle: Output of AI is only as useful as how well it can be digested by humans.

Re: Why we write our own C and C++ inference engines

#54

Post by [deleted] »

bwfan123 wrote: post #52
bastawhiz wrote: post #50

Ignoring the fact that this is clearly not written by a human, it's untrustworthy and the claims are dubious at best. 1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project. 2. The depth…

Take a look at the code. It is a conglomeration of python, rust, go, cpp etc. I was expecting lean C or C++ but all I see is a kitchen sink of code, the AI generated kind. No problem with that, but it is inconsistent with the messaging. I appreciate the intention behind this, but dont like the execution. My prediction is that this will enter the graveyard of ai generated git repos. For open-source code to get adoptio…

[deleted]

Re: Why we write our own C and C++ inference engines

#56

Post by richiejp »

bwfan123 wrote: post #52
bastawhiz wrote: post #50

Ignoring the fact that this is clearly not written by a human, it's untrustworthy and the claims are dubious at best. 1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project. 2. The depth…

Take a look at the code. It is a conglomeration of python, rust, go, cpp etc. I was expecting lean C or C++ but all I see is a kitchen sink of code, the AI generated kind. No problem with that, but it is inconsistent with the messaging. I appreciate the intention behind this, but dont like the execution. My prediction is that this will enter the graveyard of ai generated git repos. For open-source code to get adoptio…

I can only assume you are talking about LocalAI (not vllm.cpp) which is modular and contains adapters for many different AI engines written in various languages. LocalAI's core is written in pure Go and you choose what engines to install at runtime.

vLLM.cpp's runtime is pure C++, but AOT Triton compilation is done at build time in Python.

Re: Why we write our own C and C++ inference engines

#57

Post by richiejp »

bastawhiz wrote: post #50

Ignoring the fact that this is clearly not written by a human, it's untrustworthy and the claims are dubious at best. 1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project. 2. The depth…

Getting any performance improvement over vLLM is just a bonus although the RAM reductions are very nice. The main point is that we find deploying Python based projects to be a pain. I can't imagine how you know what files are unreachable in a dynamic language, but in any case that doesn't begin to solve issues like PyTorch requiring a C++ compiler at runtime to generate optimised CPU code. This isn't exactly embedded device friendly.

Having said that if you know of some magic bullet to fix the problems with the Python ecosystem (that vLLM and Pytorch are apparently unaware of) then I will be happy to try it because our CI is constantly blocked by Python related issues.

Re: Why we write our own C and C++ inference engines

#58

Post by bastawhiz »

richiejp wrote: post #57
bastawhiz wrote: post #50

Ignoring the fact that this is clearly not written by a human, it's untrustworthy and the claims are dubious at best. 1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project. 2. The depth…

Getting any performance improvement over vLLM is just a bonus although the RAM reductions are very nice. The main point is that we find deploying Python based projects to be a pain. I can't imagine how you know what files are unreachable in a dynamic language, but in any case that doesn't begin to solve issues like PyTorch requiring a C++ compiler at runtime to generate optimised CPU code. This isn't exactly embedded…

Then the article should have talked about those things and not pointed at nonsense.

Re: Why we write our own C and C++ inference engines

#59

Post by hadlock »

piterrro wrote: post #15

Could this vllm port be faster to install? Im starting gpu machine multiple times a day and it takes 5 minutes to set vllm up. If Inise this port that time is minimized?

This is on my list to evaluate, I absolutely do not want to download 9gb of supply chain risk into prod every time we upgrade, when I can compile 70mb of binary. We run vLLM in a container with hardware passthrough for gitops, having the entire environment in a single container would drastically improve things and move local LLM into a pattern that more closely follows our other CI/CD systems, rather than this hulking behemoth snowflake deployment.

Re: Why we write our own C and C++ inference engines

#60

Post by shutyourhole »

bastawhiz wrote: post #58
richiejp wrote: post #57

Earlier quoted context omitted.

Getting any performance improvement over vLLM is just a bonus although the RAM reductions are very nice. The main point is that we find deploying Python based projects to be a pain. I can't imagine how you know what files are unreachable in a dynamic language, but in any case that doesn't begin to solve issues like PyTorch requiring a C++ compiler at runtime to generate optimised CPU code. This isn't exactly embedded…

Then the article should have talked about those things and not pointed at nonsense.

[dead]
Post reply on HN