[dead]
Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
21–26 of 26 posts
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#22[dead]
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#23I am looking at a plain and simple C implemented LLM inference, and/or x86_64 assembly implemented, and/or AMD GPU RDNA assembly. Anybody?
I heard once that c++ can become assembly at some point if you type the right things in. :)
Well, the whole purpose is to be independent of invisible backdoor injectors...^W I mean compiler,
to be more accurate those compilers which deals with computer languages with an absurd and
grotesque syntax complexity.
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#24[flagged]
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#25README is in my opinion (author here) the most interesting - I wrote it to help others build useful mental model to be able to recreate the project yourself, without need to even read my code
love the details you put into to explain different techniques. it's a bit dense though, some schemas will help i think
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#26Great README. Genuinely one of the clearest walkthrough of inference internals. The KV cache section is worth lingering one as most of the OOM and throughput issues trace back to this and normally difficult to reason about. sequence length and batch size fill the cache in a way that show up under real traffic.
look forward to going over the completed course.