Not sure what's worse, if the AI slop content, or the AI code slop that makes the site consume 100% of a single CPU core (and if you enable GPU rendering on the browser, CPU usage is still significantly above zero.)
Why we write our own C and C++ inference engines
61–63 of 63 posts
Re: Why we write our own C and C++ inference engines
#62Thanks for the depth-anything.cpp port. Saves us a couple of kernel OOM's, causing connection disruptions.
Re: Why we write our own C and C++ inference engines
#63Earlier quoted context omitted.
Getting any performance improvement over vLLM is just a bonus although the RAM reductions are very nice. The main point is that we find deploying Python based projects to be a pain. I can't imagine how you know what files are unreachable in a dynamic language, but in any case that doesn't begin to solve issues like PyTorch requiring a C++ compiler at runtime to generate optimised CPU code. This isn't exactly embedded…
Then the article should have talked about those things and not pointed at nonsense.
It's literally the third paragraph:
> Deploying a Python inference stack means resolving a dependency tree at install time, on the target machine, against whatever CUDA and glibc that machine has. Deploying a ggml port means copying a shared library and a GGUF file.