31–40 of 62 posts
Earlier quoted context omitted.
Deadcode is currently working on INT4 right now on his R4D kernel. The MXFP4 fork is excellent too. Its my daily driver right now. https://codeberg.org/ggz14/radiance-vllm-mxfp4 Also has PARO quant support there too (early stage) Also speedups in both repos for 4x R9700s
That MXFP4 is an excellent project. But I have difficulties on reading that README. Is it intentionally generated like that with LLMs?
Very true, I see it for example when my co-workers apply DRY without actually thinking about the complexity it might cause. It objectively makes code smaller, but often comes with complexity cost.
Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.…
> ...one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. It makes a huge amount of sense after considering AMD's approach to graphics cards from around 2010 to 2025. They just didn't see graphics cards as viable compute platform and many who made the mistake of believing that good specs would translate into in-practice performance got bad…
I agree with your assessment that the story of supporting the competition, only to get burned, has repeated many times with AMD outside the public eye. It's why I don't put much stock in claims that things work great as long as specific flags are used.
Earlier quoted context omitted.
I'd rather buy two used rtx3090 than a single r9700 AI pro. More VRAM (some wasted due to it being non continuous), more RAM bandwidth, more aggregate compute. Only if AMD made a card like this with 48G+ I'd consider it. Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors. I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc…
2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4). And that is tight already. You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access. So surprisingly, 2x r9700 starts be a nice deal. > Also…
Earlier quoted context omitted.
2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4). And that is tight already. You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access. So surprisingly, 2x r9700 starts be a nice deal. > Also…
You don't need PCIe 5.0 x16 since RTX 30 are not PCIe 5.0 to begin with.
This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way. Also, what is the differen…
I think the piece of information that might make this click for you is the model outputs the probability distribution for all intermediate tokens even during prefill.
So for example, let's say you prefill the prompt "The quick brown fox" (and for the sake of simplicity, let's say each word is a single token). The model outputs a tensor that is [4, $vocabulary_size]. The first dimension is a token index into the input and the 2nd dimension assigns a probability to each token in the vocabulary. So even during prefill, we can look at the prediction logits for all of the intermediate tokens. That is, we can look at what the model would have predicted after "quick" and "brown", not just the tail token "fox".
In the single token autoregressive case, we just look at the next token prediction for "fox". But in the speculative decoding case we can use this information to compare the distribution of the draft model against the target model. In the greedy decoding case (ie, no sampling) we just make sure the highest probability token matches in draft and target. If we have sampling params like temperature and top-P, we have to apply something called Leviathan rejection sampling to the distribution. This basically allows us make sure the distribution is the same even if the exact probabilities are not and accept or reject draft tokens on that.
Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.…
> ...one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. It makes a huge amount of sense after considering AMD's approach to graphics cards from around 2010 to 2025. They just didn't see graphics cards as viable compute platform and many who made the mistake of believing that good specs would translate into in-practice performance got bad…