Speculative Decoding in vLLM on AMD GPUs

Post by InvectusXIV » Tue, Sep 08, 2026, 4:09 PM UTC

Post by latentsea » Wed, Sep 09, 2026, 1:44 AM UTC

nicce wrote: post #59

latentsea wrote: post #55

Earlier quoted context omitted.

> 2x RTX 3090 is not enough for proper use I don't see how. Even on 32GB you can run Q6_K_XL quant with MTP at 200k context k=q8_0, v=q5_1. So 48GB VRAM is good enough to run Q8 at long context. Also with tings like ninfer and it's various forks I'm seeing people get very good performance out of Qwen3.8 models on all sorts of NVIDIA cards.

You also want prompt/prefix cache. Otherwise there is a lot of duplicate prefill processing if you fork conversation, have a different chat window or anything like that. It therefore also makes subagents much faster.