Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speculative Decoding in vLLM on AMD GPUs

vllm.ai

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Speculative Decoding in vLLM on AMD GPUs

#41

Post by lrvick »

dist-epoch wrote: post #7
da-x wrote: post #6

Earlier quoted context omitted.

I think this has changed since then, their policies toward open source improved (e.g ROCm).

The market says the problem is still there. An NVIDIA consumer GPU sells for 50+% or more than an equivalent AMD GPU. Because people are buying NVIDIA GPUs to run local models instead of AMD ones. I did the same thing, I paid 50% more to get an 5070 Ti instead of the equivalent AMD. This is probably good for gamers, AMD GPUs are not price inflating to the same degree as NVIDIA, because they are bad at LLMs. > That wa…

> because they are bad at LLMs.

They are actually great at LLMs but you need to invest in keeping up with community tuning efforts. But I am fine with most thinking they are bad at LLMs, because I keep buying more of them!

Re: Speculative Decoding in vLLM on AMD GPUs

#43

Post by karmakaze »

intothemild wrote: post #13
androiddrew wrote: post #11

Earlier quoted context omitted.

Thankfully, there are still people willing to jump on the R9700 bandwagon and get a vLLM fork working. If you have an RDNA4 card check out https://hub.docker.com/r/stilldeadcode/vllm-radiance

Deadcode is currently working on INT4 right now on his R4D kernel. The MXFP4 fork is excellent too. Its my daily driver right now. https://codeberg.org/ggz14/radiance-vllm-mxfp4 Also has PARO quant support there too (early stage) Also speedups in both repos for 4x R9700s

The way it does tensor splitting without all-reduce cost over PCIe bus wasn't something I thought was possible.

What kind of performance are you getting with 4x R9700s--what do you do with all the VRAM (batching, concurrent requests, etc)?

Re: Speculative Decoding in vLLM on AMD GPUs

#44

Post by [deleted] »

flufluflufluffy wrote: post #16

This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way. Also, what is the differen…

[deleted]

Re: Speculative Decoding in vLLM on AMD GPUs

#45

Post by ahepp »

flufluflufluffy wrote: post #16

This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way. Also, what is the differen…

You need the next token to begin the next decode. So if you can get what might be the next token in half the time, you can kick off the next decode before the true decode for this token has finished.

If the draft was wrong you can kill the speculative decode, and you haven’t lost anything except for idle time

It’s true we can’t show the user the token until we have the true decode finished, but we can launch more work internally before we’re certain

Re: Speculative Decoding in vLLM on AMD GPUs

#46

Post by intothemild »

karmakaze wrote: post #43
intothemild wrote: post #13

Earlier quoted context omitted.

Deadcode is currently working on INT4 right now on his R4D kernel. The MXFP4 fork is excellent too. Its my daily driver right now. https://codeberg.org/ggz14/radiance-vllm-mxfp4 Also has PARO quant support there too (early stage) Also speedups in both repos for 4x R9700s

The way it does tensor splitting without all-reduce cost over PCIe bus wasn't something I thought was possible. What kind of performance are you getting with 4x R9700s--what do you do with all the VRAM (batching, concurrent requests, etc)?

personally? i have 2x gpus.. but i get bursts of ~200tok/s generation, and around 4500-5000tok/s prefil

Yeah the R4D Kernel rules imho.

Re: Speculative Decoding in vLLM on AMD GPUs

#47

Post by SomeHacker44 »

lrvick wrote: post #39
roenxi wrote: post #15

Earlier quoted context omitted.

> ...one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. It makes a huge amount of sense after considering AMD's approach to graphics cards from around 2010 to 2025. They just didn't see graphics cards as viable compute platform and many who made the mistake of believing that good specs would translate into in-practice performance got bad…

I have 4x r9700s as my coding daily drivers running qwen 3.8 27b at 80tps each. Zero complaints. Especially at $1200 each.

Please share what operating system and model runtime you use? I have two and don't get close to that with AMD's own Lemonade. Thanks!

Re: Speculative Decoding in vLLM on AMD GPUs

#48

Post by pyrolistical »

lrvick wrote: post #39
roenxi wrote: post #15

Earlier quoted context omitted.

> ...one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. It makes a huge amount of sense after considering AMD's approach to graphics cards from around 2010 to 2025. They just didn't see graphics cards as viable compute platform and many who made the mistake of believing that good specs would translate into in-practice performance got bad…

I have 4x r9700s as my coding daily drivers running qwen 3.8 27b at 80tps each. Zero complaints. Especially at $1200 each.

No way you can get that much each without extremely quants

For a single r9700 you have 637 GB/s and for qwen 3.8 27b q4_k_xl the maximum tg/s is 33 before mtp

Now if you meant 4xr9700 tensor parallelism with mtp, 80 tg/s starts to make sense

Re: Speculative Decoding in vLLM on AMD GPUs

#49

Post by suprjami »

jeanmichelselli wrote: post #23

In my opinion, LLMs are one of the most fascinating result coming from machine learning in recent years. Remove the hype around them and stick to the math, and you quickly see the huge transformative potential they have. It's great to see a lot of research being done in that direction, I wish it would mainly come from academia though..

Strange comment.

"If you remove the hype about how transformative they are, they really are transformative".

No. If you remove the hype about how transformative they are, you're left with what they actually are: a sometimes mildly useful tinker toy.

Re: Speculative Decoding in vLLM on AMD GPUs

#50

Post by esseph »

prymitive wrote: post #14
sznio wrote: post #10

Earlier quoted context omitted.

wondering when AMD will realize it can charge 2x as much for the same thing, by simply finally writing a fucking driver

The story of “great hardware ruined by poor drivers/software” is so old than its truly shocking its still a thing these days

On Linux the problem is often flipped (historically for Nvidia/AMD).

The AMD drivers are mainline kernel, while Nvidia is still handing out binary blobs.

Post reply on HN