Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
1–10 of 26 posts
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#2README is in my opinion (author here) the most interesting - I wrote it to help others build useful mental model to be able to recreate the project yourself, without need to even read my code
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#3I love the documentation formatted in lessons. I can't wait to read through it.
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#4Looks interesting, it reminds me of the first llama.cpp, but better documented.
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#5[dead]
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#6Very nice job on read me.
>>Physically, LLM is a file which contains a lot of float numbers.
aka atoms of the LLM.
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#7It seems the author believes checking the return values of CUDA API calls is not "tiny" enough :-(
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#8Very nice job on read me. >>Physically, LLM is a file which contains a lot of float numbers. aka atoms of the LLM.
the universe is just atomic if statments
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#9Wanted to add that the author has an amazing blog with lots of interesting papers: https://jedrzej.maczan.pl/
Re: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
#10README is in my opinion (author here) the most interesting - I wrote it to help others build useful mental model to be able to recreate the project yourself, without need to even read my code
Really practical teaching approach. I clicked in to see how safetensors are loaded and just kept reading. Thanks for sharing.