5.6x throughput on Kimi K2.6 by speculating less
huggingface.co
5.6x throughput on Kimi K2.6 by speculating less
1–3 of 3 posts
Re: 5.6x throughput on Kimi K2.6 by speculating less
#2hi!
open sourced a serving config for Kimi K2.6 from 90 tok/s 508 tok/s on 8xMI300X. Same weights / 0 quality loss.
Scaling is linear @15.8 tok/s per slot latency is constant. REpo has command launcher, Dockerfile, benchmark tool. Known limitations: BF16 KV only (FP8 crashes due to an AITER 384-expert constraint)
Re: 5.6x throughput on Kimi K2.6 by speculating less
#3This is a massive speed up and entirely open source!