Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We got 207 tok/s with Qwen3.5-27B on an RTX 3090

github.com

Post reply on HN
Print view View original

11–20 of 55 posts

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#11

Post by doctorpangloss »

GreenGames wrote: post #2

We built a standalone C++/ggml speculative decoder for Qwen3.5-27B Q4_K_M with a DFlash block-diffusion draft. 207.6 tok/s peak (5.46x over AR); HE 10-prompt bench averages 129.5 tok/s at DDTree budget=22, single RTX 3090, 24 GB. 3.43x over autoregressive and 2.8x over the best public SGLang AWQ number. TL;DR - Peak 207.6 tok/s DFlash vs 38.0 tok/s AR (5.46x). HE bench: 129.5 tok/s mean at DDTree budget=22. - 3.43x o…

AI authored comments are against the rules. that said what is the point of engaging here if you won't do it with your own words?

like do you understand any of what you wrote?

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#12

Post by declan_roberts »

VladVladikoff wrote: post #10
SilverElfin wrote: post #7

Why did they focus on that particular graphics card and not others, and not common laptops used by developers, or something like that?

That's a pretty popular budget friendly GPU people use for local AI, it actually seems like an excellent choice IMHO.

The only thing that compares to this is probably Mac mini with MLX models.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#13

Post by doubled112 »

VladVladikoff wrote: post #10
SilverElfin wrote: post #7

Why did they focus on that particular graphics card and not others, and not common laptops used by developers, or something like that?

That's a pretty popular budget friendly GPU people use for local AI, it actually seems like an excellent choice IMHO.

Depends on your definition of budget friendly, I suppose. I was looking around the other day and the cheapest working 24GB RTX 3090 on eBay was $1800 CAD after exchange rate, shipping and all the rest.

Hugely inflated from the $700 they were once going for. Maybe there are still deals around.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#14

Post by vessenes »

lostmsu wrote: post #8

No you did not. You got 207 tok/s on an RTX 3090 with speculative decoding which, generally speaking, is not the same quality as serving the model without it. Greedy-only decoding is even worse. There's a reason every public model comes with suggested sampling parameters. When you don't use them, output tends to degrade severely. In your case simply running a 14B model on the same hardware with the tools you compare…

why is it that speculative decoding lowers quality? My understanding of it is that you use a small/distilled fast model to predict next token - when it doesn't match, you generate more. Checking against the large model is quick.

This should maintain exactly the quality of the original model, no?

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#15

Post by tpurves »

andsoitis wrote: post #6
dirtikiti wrote: post #5

"Local AI should be a default, not a privilege: private data, no per-token bill, no vendor lock-in. The hardware to run capable models already sits on desks. The software to run those chips well doesn't." So figure out how to run it on Vulkan instead of requiring the user to be locked into expensive CUDA cards.

Why doesn’t Apple?

Like with all new tech trends, it takes them a hot minute to catch up, but it's highly likely they will (eventually) release some killer platforms for local AI. The shared memory, high bandwidth and power-efficiency of their M chips is a near-ideal architecture. If/when they finally push out the M5-ultra, that could be round one (albeit still not at the best price/performace vs comparable cloud api tokens). A real mass-market killer device for local LLMs is still going to require some remediation of the global DRAM shortages, and maybe the M6/M7 generation.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#16

Post by Aurornis »

This is a Claude-code generated repo that implements some ideas from research papers. If you follow this space, every paper release spawns tens or hundreds of vibecoded repos like this that get spammed to Reddit, Hacker News, and other sites.

It's generally best to overlook the vibecoded repos and go closer to the source for up to date information. In this case, z-lab already showed Qwen3.5-27B with DFlash last month: https://huggingface.co/z-lab/Qwen3.5-27B-DFlash

This repo is an example of what you get if you point Claude Code at the upstream repo and have it iterate with some other objective (loading GGUF). They also included DDTree in there somewhere.

You also need to look closely at the claims. A classic trick in these repos is to cherry-pick numbers that make the work in the repo look extraordinary until you start reading the details. From my quick read, this repo is using Q4 quantization on the KV cache which does not produce good results. Someone who reads everything in detail might find more tricks. This is par for all of these demo repos because the goal is to impress casual viewers with big numbers.

I'm trying to find where they get the 207 tok/s number but the 207 number only appears in their headline claim. If you read deeper the real numbers are half that or less.

There are also several (possibly vibecoded, I haven't checked) draft PRs and forks to use these techniques on upstream llama.cpp that would be much more useful for experimenting. One example I picked at random: https://github.com/ggml-org/llama.cpp/pull/22105

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#17

Post by kingstnap »

lostmsu wrote: post #8

No you did not. You got 207 tok/s on an RTX 3090 with speculative decoding which, generally speaking, is not the same quality as serving the model without it. Greedy-only decoding is even worse. There's a reason every public model comes with suggested sampling parameters. When you don't use them, output tends to degrade severely. In your case simply running a 14B model on the same hardware with the tools you compare…

Speculative decoding doesn't degrade output quality. The distribution it produces is exactly the same if you do it correctly. The original paper on it clearly talks about this. [0]

Speculative decoding is the same as speculative execution on CPUs. As long as you walk back on an incorrect prediction (i.e. the speculated tokens weren't accepted) then everything is mathematically exactly the same. It just uses more parallelism (specificslly higher arithmetic intensity).

[0] https://arxiv.org/abs/2211.17192

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#18

Post by Aurornis »

SilverElfin wrote: post #7

Why did they focus on that particular graphics card and not others, and not common laptops used by developers, or something like that?

The repo is very vibecoded (Claude is co-author, READMEs are obviously AI).

This is the output of someone with a 3090 who pointed Claude Code at some research papers and possibly the upstream implementations of these techniques and then posted the output as original work.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#19

Post by dvt »

andsoitis wrote: post #6
dirtikiti wrote: post #5

"Local AI should be a default, not a privilege: private data, no per-token bill, no vendor lock-in. The hardware to run capable models already sits on desks. The software to run those chips well doesn't." So figure out how to run it on Vulkan instead of requiring the user to be locked into expensive CUDA cards.

Why doesn’t Apple?

Apple has Metal, which is already pretty well-integrated in llama.cpp, various Python libs, and mistral-rs & candle. Unpopular opinion, but Vulkan is hot garbage and the definition of "design by committee." There's a reason people still prefer CUDA, whereas most code could likely be programmatically ported anyway.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#20

Post by Aurornis »

dirtikiti wrote: post #5

"Local AI should be a default, not a privilege: private data, no per-token bill, no vendor lock-in. The hardware to run capable models already sits on desks. The software to run those chips well doesn't." So figure out how to run it on Vulkan instead of requiring the user to be locked into expensive CUDA cards.

So everyone is aware, you can already run Qwen3.5-27B on Vulkan or Apple's hardware. Every major inference engine supports it right now.

This repo is a vibecoded demo implementation of some recent research papers combined with some optimizations that sacrifice quality for speed to get a big number that looks impressive. The 207 tok/s number they're claiming only appears in the headline. The results they show are half that or less, so I already don't trust anything they're saying they accomplished.

If you want to run Qwen3.5-27B you can do it with a project llama.cpp on CUDA, Vulkan, Apple, or even CPU.

Post reply on HN