Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025) • Hacker News Board

Re: Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)

#11

Post by mmastrac » Fri, Aug 07, 2026, 12:43 PM UTC

brainless wrote: post #10

mmastrac wrote: post #6

Earlier quoted context omitted.

I've been working on a fresh, AI assisted port of DiffusionGemma from scratch and it takes a significant amount of time to deslop. I've spend a nonzero amount of time on refactoring and comment-vomit cleanup. https://github.com/mmastrac/diffgemma

Any plans to support smaller models? I have a M4 Mac Mini with 16GB unified memory and an RTX 3060 (Laptop) with 6GB VRAM. My own product experiments all revolve around small models and harness around them. Happy to contribute.

I've been pondering a smaller quantization and experts swapping for this! Happy to take on PRs if you want to experiment as well.

Re: Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)

#12

Post by bytepoet » Fri, Aug 07, 2026, 3:51 PM UTC

yu3zhou4 wrote: post #9

miki123211 wrote: post #3

Another great way to understand how vllm works is to read the code of nano-vllm[1]. It's basically "vllm but cut down to size. It's ~5kloc, supports just one model, disposes of some of the abstraction layers that vllm needs due to its codebase size, but contains all the major pieces that make an inference engine fast. [1] https://github.com/GeeeekExplorer/nano-vllm

If you prefer C++ and CUDA, then there's also tiny-vllm of mine [0] - recently we broke 1k gh stars [0] https://github.com/jmaczan/tiny-vllm

Thanks for tiny-vllm! The documentation is amazing. I'm using it as a resource for a university course I'm teaching.

Re: Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)

#13

Post by yu3zhou4 » Sat, Aug 08, 2026, 3:24 PM UTC

bytepoet wrote: post #12

yu3zhou4 wrote: post #9

Earlier quoted context omitted.

If you prefer C++ and CUDA, then there's also tiny-vllm of mine [0] - recently we broke 1k gh stars [0] https://github.com/jmaczan/tiny-vllm

Thanks for tiny-vllm! The documentation is amazing. I'm using it as a resource for a university course I'm teaching.

I’m happy it helps you!