Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speculative Decoding in vLLM on AMD GPUs

vllm.ai

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Speculative Decoding in vLLM on AMD GPUs

#51

Post by esseph »

numpad0 wrote: post #22
sznio wrote: post #10

Earlier quoted context omitted.

wondering when AMD will realize it can charge 2x as much for the same thing, by simply finally writing a fucking driver

AMD also deprecates cards and throw in wrenches for cards that are hot in used markets. Latest ROCm kind of works on MI50 but requisite files are taken out of just standard Ubuntu installation. They truly don't understand marketing.

That is a ~9 year old card. I'm not sure any of the newer work AMD has been doing can utilize that hardware at all.

It would have been sold off to grey market or destroyed 3-4yr ago in most companies.

Newer, much cheaper stuff ironically would run better. That card isn't even really designed for AI workloads at all.

Re: Speculative Decoding in vLLM on AMD GPUs

#52

Post by naasking »

pyrolistical wrote: post #48
lrvick wrote: post #39

Earlier quoted context omitted.

I have 4x r9700s as my coding daily drivers running qwen 3.8 27b at 80tps each. Zero complaints. Especially at $1200 each.

No way you can get that much each without extremely quants For a single r9700 you have 637 GB/s and for qwen 3.8 27b q4_k_xl the maximum tg/s is 33 before mtp Now if you meant 4xr9700 tensor parallelism with mtp, 80 tg/s starts to make sense

Q4 isn't an extreme quant, and I average 75 toks/s on code, 45 tok/s on prose with MTP.

Re: Speculative Decoding in vLLM on AMD GPUs

#53

Post by [deleted] »

dist-epoch wrote: post #5
intothemild wrote: post #2

Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.…

George Hotz in June 2023: > I have had direct contact with members of the AMD RTG team and I was disgusted to find that AMD doesn't even provide them with hardware to work on. The developer I was working with had to buy the GPU he was writing drivers for.

[deleted]

Re: Speculative Decoding in vLLM on AMD GPUs

#54

Post by latentsea »

pyrolistical wrote: post #48
lrvick wrote: post #39

Earlier quoted context omitted.

I have 4x r9700s as my coding daily drivers running qwen 3.8 27b at 80tps each. Zero complaints. Especially at $1200 each.

No way you can get that much each without extremely quants For a single r9700 you have 637 GB/s and for qwen 3.8 27b q4_k_xl the maximum tg/s is 33 before mtp Now if you meant 4xr9700 tensor parallelism with mtp, 80 tg/s starts to make sense

You can get those numbers with https://codeberg.org/ggz14/radiance-vllm-mxfp4. I also can get it on a single R9700 but the 75 ~ 80 t/s is only peak acceptance of very predictable tokens like coding or json, and averages lower for prose. It's still much faster than regular llama.cpp.

Re: Speculative Decoding in vLLM on AMD GPUs

#55

Post by latentsea »

nicce wrote: post #28
Roark66 wrote: post #18

Earlier quoted context omitted.

I'd rather buy two used rtx3090 than a single r9700 AI pro. More VRAM (some wasted due to it being non continuous), more RAM bandwidth, more aggregate compute. Only if AMD made a card like this with 48G+ I'd consider it. Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors. I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc…

2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4). And that is tight already. You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access. So surprisingly, 2x r9700 starts be a nice deal. > Also…

> 2x RTX 3090 is not enough for proper use

I don't see how. Even on 32GB you can run Q6_K_XL quant with MTP at 200k context k=q8_0, v=q5_1. So 48GB VRAM is good enough to run Q8 at long context. Also with tings like ninfer and it's various forks I'm seeing people get very good performance out of Qwen3.8 models on all sorts of NVIDIA cards.

Re: Speculative Decoding in vLLM on AMD GPUs

#56

Post by [deleted] »

suprjami wrote: post #49
jeanmichelselli wrote: post #23

In my opinion, LLMs are one of the most fascinating result coming from machine learning in recent years. Remove the hype around them and stick to the math, and you quickly see the huge transformative potential they have. It's great to see a lot of research being done in that direction, I wish it would mainly come from academia though..

Strange comment. "If you remove the hype about how transformative they are, they really are transformative". No. If you remove the hype about how transformative they are, you're left with what they actually are: a sometimes mildly useful tinker toy.

[deleted]

Re: Speculative Decoding in vLLM on AMD GPUs

#57

Post by lrvick »

SomeHacker44 wrote: post #47
lrvick wrote: post #39

Earlier quoted context omitted.

I have 4x r9700s as my coding daily drivers running qwen 3.8 27b at 80tps each. Zero complaints. Especially at $1200 each.

Please share what operating system and model runtime you use? I have two and don't get close to that with AMD's own Lemonade. Thanks!

Lemonade is one of the worst performing options. Run any modern Linux distro and ask your current LLM to setup llama.cpp with dflash2 for you as an unprivileged container running from a systemd user unit.

Obviously only on a system you do not trust at all.

Re: Speculative Decoding in vLLM on AMD GPUs

#58

Post by karmakaze »

intothemild wrote: post #46
karmakaze wrote: post #43

Earlier quoted context omitted.

The way it does tensor splitting without all-reduce cost over PCIe bus wasn't something I thought was possible. What kind of performance are you getting with 4x R9700s--what do you do with all the VRAM (batching, concurrent requests, etc)?

personally? i have 2x gpus.. but i get bursts of ~200tok/s generation, and around 4500-5000tok/s prefil Yeah the R4D Kernel rules imho.

Similar here peak ~250 and down to ~120 as it gets close to 128k (which is where I set DSH compaction) though it can readily do 256k.

I just got DeepSeek Harness (DSH) set up with 2x R9700 and it's rather mind blowing that these can do actual work and quickly. Up until now I've always been evaluating and searching for better hardware/model/tweaks. This is much more than I even hoped for and considered getting extra 3090/4090. Now I can stop looking/tweaking and start using it for all the different things I've yet to discover it's good for. I do plan to also try/use Hermes and Pi. DSH is annoying that every plugin install/remove requires a restart--given that "everything's a plugin".

Re: Speculative Decoding in vLLM on AMD GPUs

#59

Post by nicce »

latentsea wrote: post #55
nicce wrote: post #28

Earlier quoted context omitted.

2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4). And that is tight already. You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access. So surprisingly, 2x r9700 starts be a nice deal. > Also…

> 2x RTX 3090 is not enough for proper use I don't see how. Even on 32GB you can run Q6_K_XL quant with MTP at 200k context k=q8_0, v=q5_1. So 48GB VRAM is good enough to run Q8 at long context. Also with tings like ninfer and it's various forks I'm seeing people get very good performance out of Qwen3.8 models on all sorts of NVIDIA cards.

You also want prompt/prefix cache. Otherwise there is a lot of duplicate prefill processing if you fork conversation, have a different chat window or anything like that. It therefore also makes subagents much faster.

Re: Speculative Decoding in vLLM on AMD GPUs

#60

Post by 3abiton »

intothemild wrote: post #2

Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.…

Honestly a big part of this is AMD's lackluster strategy to GPU software. To say it's lacking is understatement. At least for non-data center gpus.
Post reply on HN