Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We got 207 tok/s with Qwen3.5-27B on an RTX 3090

github.com

Post reply on HN
Print view View original

21–30 of 55 posts

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#21

Post by nodja »

lostmsu wrote: post #8

No you did not. You got 207 tok/s on an RTX 3090 with speculative decoding which, generally speaking, is not the same quality as serving the model without it. Greedy-only decoding is even worse. There's a reason every public model comes with suggested sampling parameters. When you don't use them, output tends to degrade severely. In your case simply running a 14B model on the same hardware with the tools you compare…

> speculative decoding which, generally speaking, is not the same quality as serving the model without it.

I've never heard of ANY speculative decoding that wasn't lossless. If it was lossy it'd be called something else.

This page is just a port of DFLASH to gguf format, it only implements greedy decoding like you said so the outputs will be inferior, but not inferior to greedy decoding on the original model. Tho that's just a matter of implementing temperature, top_k, etc.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#22

Post by coldtea »

andsoitis wrote: post #6
dirtikiti wrote: post #5

"Local AI should be a default, not a privilege: private data, no per-token bill, no vendor lock-in. The hardware to run capable models already sits on desks. The software to run those chips well doesn't." So figure out how to run it on Vulkan instead of requiring the user to be locked into expensive CUDA cards.

Why doesn’t Apple?

Vulkan is not Apple.

Metal is Apple's API.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#23

Post by lostmsu »

vessenes wrote: post #14
lostmsu wrote: post #8

No you did not. You got 207 tok/s on an RTX 3090 with speculative decoding which, generally speaking, is not the same quality as serving the model without it. Greedy-only decoding is even worse. There's a reason every public model comes with suggested sampling parameters. When you don't use them, output tends to degrade severely. In your case simply running a 14B model on the same hardware with the tools you compare…

why is it that speculative decoding lowers quality? My understanding of it is that you use a small/distilled fast model to predict next token - when it doesn't match, you generate more. Checking against the large model is quick. This should maintain exactly the quality of the original model, no?

I looked up, and you are correct in regards to the specific algorithm used. In general there are approximate algorithms for speculative decoding.

Greedy decoding means it is still not ready though.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#25

Post by j45 »

Aurornis wrote: post #16

This is a Claude-code generated repo that implements some ideas from research papers. If you follow this space, every paper release spawns tens or hundreds of vibecoded repos like this that get spammed to Reddit, Hacker News, and other sites. It's generally best to overlook the vibecoded repos and go closer to the source for up to date information. In this case, z-lab already showed Qwen3.5-27B with DFlash last month…

Appreciate the reading and things to go learn more from.

Learning about Qwen 3.5, and also learning how Gemma 4 appears to be unique (relatively speaking), and Apple possibly using some type of Gemma model on-device I think will also help fill in how to track local model and local device capabilities which could be additional measures/KPIs as well.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#26

Post by dist-epoch »

doubled112 wrote: post #13
VladVladikoff wrote: post #10

Earlier quoted context omitted.

That's a pretty popular budget friendly GPU people use for local AI, it actually seems like an excellent choice IMHO.

Depends on your definition of budget friendly, I suppose. I was looking around the other day and the cheapest working 24GB RTX 3090 on eBay was $1800 CAD after exchange rate, shipping and all the rest. Hugely inflated from the $700 they were once going for. Maybe there are still deals around.

Because they are hugely more useful now than running some stupid game at 240 fps instead of 60 fps.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#27

Post by GreenGames »

Aurornis wrote: post #16

This is a Claude-code generated repo that implements some ideas from research papers. If you follow this space, every paper release spawns tens or hundreds of vibecoded repos like this that get spammed to Reddit, Hacker News, and other sites. It's generally best to overlook the vibecoded repos and go closer to the source for up to date information. In this case, z-lab already showed Qwen3.5-27B with DFlash last month…

This reads like you didn’t read the post.

z-lab runs BF16 on B200 (54+ GB). There is no z-lab path that fits on a 24 GB 3090. That is literally the entire point of our work, and it is stated in the second paragraph. If you had checked the HF model card you linked before posting, you would see the same thing. Before this repo, there was no path to run this... SGLang's GGUF path for this model is broken. llama.cpp doesn't have DFlash speculative decoding at all. If you wanted to run this hybrid model fast on a 24 GB consumer card, there was nothing...

That took weeks of real engineering.

Calling that "vibecoded" because we used a bit of AI in the README is clean is the laziest possible critique. An LLM reading the DFlash paper does not catch verify_logits_buf being sized vocabq_len when DDTree reads vocab(budget+1). That is hours of debugging with nvidia-smi and memory sanitizers, not prompting.

The 207 and 129.5 numbers are both in the second sentence of the post and again in the TL;DR. 207.6 is peak tok/s in the linked demo video, 129.5 is the HumanEval 10-prompt mean at DDTree budget=22. We specify both just behind the title.

On the Q4 KV cache: the tradeoff is disclosed with actual numbers. AL 8.56 -> 8.33 at short context (3% drop), dramatically better at long context. It’s the only way 128K allocates on 24 GB. The binary is env-selectable, you can run BF16 KV if you don’t need 128K. Both are benchmarked.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#29

Post by ndriscoll »

vessenes wrote: post #14
lostmsu wrote: post #8

No you did not. You got 207 tok/s on an RTX 3090 with speculative decoding which, generally speaking, is not the same quality as serving the model without it. Greedy-only decoding is even worse. There's a reason every public model comes with suggested sampling parameters. When you don't use them, output tends to degrade severely. In your case simply running a 14B model on the same hardware with the tools you compare…

why is it that speculative decoding lowers quality? My understanding of it is that you use a small/distilled fast model to predict next token - when it doesn't match, you generate more. Checking against the large model is quick. This should maintain exactly the quality of the original model, no?

AFAIU It's not that checking against the large model is quick (in the usual P!=NP sense that checking an answer is easier than finding one). It's that you can batch your checks. So you speculate the next 5 tokens, and then you can parallelize the large model running once for the batch of [...,n+1], [...,n+2], [...,n+3], [...,n+4], [...,n+5]. If you guessed right for a prefix, you turned a sequential problem (computing next token from current prefix) into a parallel one (doing multiple prefixes together) that the GPU likes. If you guessed wrong, you have to throw away the suffix starting at the wrong guess, and you wasted some extra energy computing.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#30

Post by tempaccount5050 »

doctorpangloss wrote: post #11
GreenGames wrote: post #2

We built a standalone C++/ggml speculative decoder for Qwen3.5-27B Q4_K_M with a DFlash block-diffusion draft. 207.6 tok/s peak (5.46x over AR); HE 10-prompt bench averages 129.5 tok/s at DDTree budget=22, single RTX 3090, 24 GB. 3.43x over autoregressive and 2.8x over the best public SGLang AWQ number. TL;DR - Peak 207.6 tok/s DFlash vs 38.0 tok/s AR (5.46x). HE bench: 129.5 tok/s mean at DDTree budget=22. - 3.43x o…

AI authored comments are against the rules. that said what is the point of engaging here if you won't do it with your own words? like do you understand any of what you wrote?

I find these comments hilarious. Are we supposed to build AI and then not use it? Super goofy.
Post reply on HN