I've been building streaming inference kernels for Zonos, the open SSM-based (Mamba2) TTS model, on an A10G. This post is two hand-written CuTeDSL kernels in two regimes, and the contrast is the point.

The first is a skinny bf16 GEMV for the batch-1 decode projections. It beats cuBLAS by up to 1.36x, not because it's clever, but because cuBLAS pads the tiny M dimension out to a tensor-core GEMM tile and leaves memory bandwidth on the floor at exactly that shape. It's memory-bound, so I measured the actual achievable read bandwidth (520 GB/s, not the 600 GB/s spec) and tuned to that roofline.

The second is a from-scratch port of Mamba2's four-stage SSD chunked scan. It loses to Tri Dao's Triton by ~2x, and the gap widens with sequence length. Triton is nearly flat as the work grows 32x while mine climbs linearly, which is the signature of exposed per-chunk load latency that Triton's software pipelining hides and my single-buffered version doesn't.

The takeaway I keep coming back to: a hand kernel wins where the library is forced to generalize and loses where the library is already specialized for your exact problem. The whole skill is knowing which situation you're in before you spend the three weeks.

Also documented: a kernel that benchmarked at 50x the chip's memory bandwidth (a CUDA-graph stream-capture bug), and why a bit-accurate drop-in kernel still changes 4.6% of tokens in a 34-layer autoregressive model.

Writeup: https://kyrieblunders.bearblog.dev/the-kernels-that-were-the... Code and profiling docs: https://github.com/Vishal-Padia/Sonata

Happy to answer questions.

Show HN: Where a hand-written GPU kernel beats the library, and where it can't
kyrieblunders.bearblog.dev