Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speculative Decoding in vLLM on AMD GPUs

vllm.ai

Post reply on HN
Print view View original

11–20 of 62 posts

Re: Speculative Decoding in vLLM on AMD GPUs

#11

Post by androiddrew »

intothemild wrote: post #2

Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.…

Thankfully, there are still people willing to jump on the R9700 bandwagon and get a vLLM fork working. If you have an RDNA4 card check out https://hub.docker.com/r/stilldeadcode/vllm-radiance

Re: Speculative Decoding in vLLM on AMD GPUs

#12

Post by esseph »

dist-epoch wrote: post #7
da-x wrote: post #6

Earlier quoted context omitted.

I think this has changed since then, their policies toward open source improved (e.g ROCm).

The market says the problem is still there. An NVIDIA consumer GPU sells for 50+% or more than an equivalent AMD GPU. Because people are buying NVIDIA GPUs to run local models instead of AMD ones. I did the same thing, I paid 50% more to get an 5070 Ti instead of the equivalent AMD. This is probably good for gamers, AMD GPUs are not price inflating to the same degree as NVIDIA, because they are bad at LLMs. > That wa…

> AMD GPUs are not price inflating to the same degree as NVIDIA, because they are bad at LLMs.

9060 XT 16GB seems to have some great performance with gpt oss 20B and others, and works great with their lemonade-server.

https://lemonade-server.ai/

What exactly do you think is running on a Strix Halo?

Re: Speculative Decoding in vLLM on AMD GPUs

#13

Post by intothemild »

androiddrew wrote: post #11
intothemild wrote: post #2

Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.…

Thankfully, there are still people willing to jump on the R9700 bandwagon and get a vLLM fork working. If you have an RDNA4 card check out https://hub.docker.com/r/stilldeadcode/vllm-radiance

Deadcode is currently working on INT4 right now on his R4D kernel.

The MXFP4 fork is excellent too. Its my daily driver right now. https://codeberg.org/ggz14/radiance-vllm-mxfp4

Also has PARO quant support there too (early stage)

Also speedups in both repos for 4x R9700s

Re: Speculative Decoding in vLLM on AMD GPUs

#14

Post by prymitive »

sznio wrote: post #10
dist-epoch wrote: post #7

Earlier quoted context omitted.

The market says the problem is still there. An NVIDIA consumer GPU sells for 50+% or more than an equivalent AMD GPU. Because people are buying NVIDIA GPUs to run local models instead of AMD ones. I did the same thing, I paid 50% more to get an 5070 Ti instead of the equivalent AMD. This is probably good for gamers, AMD GPUs are not price inflating to the same degree as NVIDIA, because they are bad at LLMs. > That wa…

wondering when AMD will realize it can charge 2x as much for the same thing, by simply finally writing a fucking driver

The story of “great hardware ruined by poor drivers/software” is so old than its truly shocking its still a thing these days

Re: Speculative Decoding in vLLM on AMD GPUs

#15

Post by roenxi »

intothemild wrote: post #2

Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.…

> ...one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored.

It makes a huge amount of sense after considering AMD's approach to graphics cards from around 2010 to 2025. They just didn't see graphics cards as viable compute platform and many who made the mistake of believing that good specs would translate into in-practice performance got badly burned. I'd have been involved in the AI boom but for an expensive AMD graphics card, I'm not going to forget that for a while.

George Hotz was interesting as a public example, but I think his story probably repeated a few times outside the public eye. People tried to make AMD work and ended up the worse for it.

People who had an interest in using AMD cards to get things done are probably by and large waiting for a new generation of hopefuls to prove this time is different. The mutterings out of AMD are promising, but that isn't persuasive enough given the scale of the failures.

Re: Speculative Decoding in vLLM on AMD GPUs

#16

Post by flufluflufluffy »

This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way.

Also, what is the difference between “target model” and “target-model,” if any? I feel like half the instances of that phrase included the hyphen and half didn’t.

Re: Speculative Decoding in vLLM on AMD GPUs

#17

Post by lucrbvi »

flufluflufluffy wrote: post #16

This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way. Also, what is the differen…

The target model is the original LLM that is large and expensive. It can verify candidate tokens in a single forward pass. It means you give all the context + candidate tokens that passes in parallel in the backbone, then you pass the language head (a matmul transformation to produce the token distribution) on all the candidate tokens and you can keep or drop tokens based on how many "quality" you want.

Re: Speculative Decoding in vLLM on AMD GPUs

#18

Post by Roark66 »

intothemild wrote: post #2

Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.…

I'd rather buy two used rtx3090 than a single r9700 AI pro. More VRAM (some wasted due to it being non continuous), more RAM bandwidth, more aggregate compute.

Only if AMD made a card like this with 48G+ I'd consider it.

Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.

I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc b70.

I was actually very interested in this so I checked the details. Turns out it was 200 simultaneous users running the same 1024 token prompt :D so all the experts got maximum parallelism.

How often are you going to run 200 parallel sessions with a tiny context and same prompt running at 7tok/s.

Based on how much my rtx3090 is getting on a single user (150tok/s) I'm estimating b70 to probably get less than that.

Sadly nvidia is king now.

Also, most of us already have nvidia cards and no inference software supports mixing let's say nvidia, Intel and amd cards in inference of one model.

Re: Speculative Decoding in vLLM on AMD GPUs

#19

Post by brookst »

_factor wrote: post #9
websap wrote: post #4

Earlier quoted context omitted.

Yeah, as a business AMD should first care about getting their DC grade hardware optimized for inference workloads. It's unfortunate that most of HN discussion has devolved to me-ish.

Then they should stop selling hardware they don’t plan to support. Me-ish when you spend $1,500 on a piece of hardware is completely acceptable.

Never buy hardware based on expectations of future features, especially if there’s no promise from the vendor.

Re: Speculative Decoding in vLLM on AMD GPUs

#20

Post by mrhenio »

websap wrote: post #4
minraws wrote: post #3

Earlier quoted context omitted.

I don't see a reason why it should AMD doesn't care about lower end prosumers atm. They might in the future but future is in the future ofc Edit: to be clear I think it's ridiculous they don't but from a company's stand point it doesn't make much sense

Yeah, as a business AMD should first care about getting their DC grade hardware optimized for inference workloads. It's unfortunate that most of HN discussion has devolved to me-ish.

[flagged]
Post reply on HN