Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speculative Decoding in vLLM on AMD GPUs

vllm.ai

Post reply on HN
Print view View original

21–30 of 62 posts

Re: Speculative Decoding in vLLM on AMD GPUs

#21

Post by [deleted] »

lucrbvi wrote: post #17
flufluflufluffy wrote: post #16

This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way. Also, what is the differen…

The target model is the original LLM that is large and expensive. It can verify candidate tokens in a single forward pass. It means you give all the context + candidate tokens that passes in parallel in the backbone, then you pass the language head (a matmul transformation to produce the token distribution) on all the candidate tokens and you can keep or drop tokens based on how many "quality" you want.

[deleted]

Re: Speculative Decoding in vLLM on AMD GPUs

#22

Post by numpad0 »

sznio wrote: post #10
dist-epoch wrote: post #7

Earlier quoted context omitted.

The market says the problem is still there. An NVIDIA consumer GPU sells for 50+% or more than an equivalent AMD GPU. Because people are buying NVIDIA GPUs to run local models instead of AMD ones. I did the same thing, I paid 50% more to get an 5070 Ti instead of the equivalent AMD. This is probably good for gamers, AMD GPUs are not price inflating to the same degree as NVIDIA, because they are bad at LLMs. > That wa…

wondering when AMD will realize it can charge 2x as much for the same thing, by simply finally writing a fucking driver

AMD also deprecates cards and throw in wrenches for cards that are hot in used markets. Latest ROCm kind of works on MI50 but requisite files are taken out of just standard Ubuntu installation. They truly don't understand marketing.

Re: Speculative Decoding in vLLM on AMD GPUs

#23

Post by jeanmichelselli »

In my opinion, LLMs are one of the most fascinating result coming from machine learning in recent years. Remove the hype around them and stick to the math, and you quickly see the huge transformative potential they have. It's great to see a lot of research being done in that direction, I wish it would mainly come from academia though..

Re: Speculative Decoding in vLLM on AMD GPUs

#25

Post by daemonologist »

flufluflufluffy wrote: post #16

This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way. Also, what is the differen…

You're correct that it needs to run the full model to "verify" a token, but LLM inference benefits from batching - it's much faster to run twice in parallel than sequentially. So the draft model runs ~2-5 tokens ahead, and the full model then runs ~3-6 batches in parallel using those tokens, and can skip ahead by however many results match.

If you have some other source of parallel data (lots of users, many separate tasks) then speculative decoding might not provide any benefit.

Re: Speculative Decoding in vLLM on AMD GPUs

#26

Post by porridgeraisin »

flufluflufluffy wrote: post #16

This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way. Also, what is the differen…

> it must perform it's normal autoregressive decoding to know what is the correct token in order to have something to compare with

Correct except for the word "autoregressive". When you have to verify a sequence of tokens (which were autoregressively generated by the cheap model), you can do each token in parallel. This amortizes the cost of loading the weights from vram to the processors (the primary cost in LLM serving) across those tokens. Cost here is wall clock time, as well as power.

The autoregressive decoding that generates this batch of tokens is delegated to the cheaper model where the cost of loading the weights is lower and so not amortizing it is fine.

Verification means, how close is each token in this sequence to the one I would have output. You keep the longest prefix that is close enough for your liking.

Re: Speculative Decoding in vLLM on AMD GPUs

#27

Post by karmakaze »

intothemild wrote: post #13
androiddrew wrote: post #11

Earlier quoted context omitted.

Thankfully, there are still people willing to jump on the R9700 bandwagon and get a vLLM fork working. If you have an RDNA4 card check out https://hub.docker.com/r/stilldeadcode/vllm-radiance

Deadcode is currently working on INT4 right now on his R4D kernel. The MXFP4 fork is excellent too. Its my daily driver right now. https://codeberg.org/ggz14/radiance-vllm-mxfp4 Also has PARO quant support there too (early stage) Also speedups in both repos for 4x R9700s

Thanks! Didn't expect to see this here. Exactly what I needed to run Qwen3.8-27B-Quark-AWQ-MXFP4-native.gguf as well as other experiments on one or 2x R9700's (I hope).

Re: Speculative Decoding in vLLM on AMD GPUs

#28

Post by nicce »

Roark66 wrote: post #18
intothemild wrote: post #2

Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.…

I'd rather buy two used rtx3090 than a single r9700 AI pro. More VRAM (some wasted due to it being non continuous), more RAM bandwidth, more aggregate compute. Only if AMD made a card like this with 48G+ I'd consider it. Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors. I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc…

2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4). And that is tight already.

You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access.

So surprisingly, 2x r9700 starts be a nice deal.

> Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.

Well, luckily these are not vendor numbers. Prefill also scales almost linearly with the amount of GPUs.

Re: Speculative Decoding in vLLM on AMD GPUs

#29

Post by nicce »

intothemild wrote: post #13
androiddrew wrote: post #11

Earlier quoted context omitted.

Thankfully, there are still people willing to jump on the R9700 bandwagon and get a vLLM fork working. If you have an RDNA4 card check out https://hub.docker.com/r/stilldeadcode/vllm-radiance

Deadcode is currently working on INT4 right now on his R4D kernel. The MXFP4 fork is excellent too. Its my daily driver right now. https://codeberg.org/ggz14/radiance-vllm-mxfp4 Also has PARO quant support there too (early stage) Also speedups in both repos for 4x R9700s

That MXFP4 is an excellent project. But I have difficulties on reading that README. Is it intentionally generated like that with LLMs?

Re: Speculative Decoding in vLLM on AMD GPUs

#30

Post by cesarb »

flufluflufluffy wrote: post #16

This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way. Also, what is the differen…

> Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with.

Yes, but you can do it in parallel.

Suppose you predicted the tokens "D E F" in the sequence "A B C D E F". To "generate" the last token (F), it must know all preceding tokens (A B C D E). To "generate" the next-to-last token (E), it must know all preceding tokens (A B C D). And so on.

Assuming the prediction is correct, it can then run the "generation" for tokens D, E, and F at the same time. At the end, after all these tokens were "generated", it compares each token with the prediction; if the "generation" result was "D H F" it knows it has to discard the last two predicted tokens (and output "D H"), if the "generation" was "D E H" it knows it has to discard the last predicted token (and output "D E H"), etc.

And the most important part is that you can do it in parallel for each layer of the model. That is, you run "A B C D E F" through the first layer, then through the second layer, and so on; you only have to load the model weights from memory once for each layer. Instead of reading the full weights for all layers once for D, then once for E, then once for F, you only read them once for "D E F", and if the prediction was correct, you output three tokens by the (memory read) price of one (you still had to do the same amount of compute, but AFAIK LLMs tend to be more memory-bound than compute-bound).

Post reply on HN