Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We got 207 tok/s with Qwen3.5-27B on an RTX 3090

github.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#31

Post by refulgentis »

GreenGames wrote: post #27
Aurornis wrote: post #16

This is a Claude-code generated repo that implements some ideas from research papers. If you follow this space, every paper release spawns tens or hundreds of vibecoded repos like this that get spammed to Reddit, Hacker News, and other sites. It's generally best to overlook the vibecoded repos and go closer to the source for up to date information. In this case, z-lab already showed Qwen3.5-27B with DFlash last month…

This reads like you didn’t read the post. z-lab runs BF16 on B200 (54+ GB). There is no z-lab path that fits on a 24 GB 3090. That is literally the entire point of our work, and it is stated in the second paragraph. If you had checked the HF model card you linked before posting, you would see the same thing. Before this repo, there was no path to run this... SGLang's GGUF path for this model is broken. llama.cpp does…

You wrote this reply with Claude, and it's lying about it only being README.md. OP, and I, know this because you and Claude documented it.*

I use the same tools, I'm not mad at you for using it. It's just, idk man, you want to use it tactically in ways that are a net benefit to you. Not in ways that embarrass you or lie.

* https://github.com/Luce-Org/lucebox-hub/commit/cfc38f67275ee...

* * Here's Claude's version of this very post if you want to see an example of Claude voice vs. original and how to spot it: https://gist.githubusercontent.com/jpohhhh/a42060f0f34339c4b...

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#32

Post by Grimblewald »

Aurornis wrote: post #20
dirtikiti wrote: post #5

"Local AI should be a default, not a privilege: private data, no per-token bill, no vendor lock-in. The hardware to run capable models already sits on desks. The software to run those chips well doesn't." So figure out how to run it on Vulkan instead of requiring the user to be locked into expensive CUDA cards.

So everyone is aware, you can already run Qwen3.5-27B on Vulkan or Apple's hardware. Every major inference engine supports it right now. This repo is a vibecoded demo implementation of some recent research papers combined with some optimizations that sacrifice quality for speed to get a big number that looks impressive. The 207 tok/s number they're claiming only appears in the headline. The results they show are half…

This, even on android via termux you can run ollama with gpu accelaration on phone. This works, though milage will vary.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#33

Post by zdragnar »

tempaccount5050 wrote: post #30
doctorpangloss wrote: post #11

Earlier quoted context omitted.

AI authored comments are against the rules. that said what is the point of engaging here if you won't do it with your own words? like do you understand any of what you wrote?

I find these comments hilarious. Are we supposed to build AI and then not use it? Super goofy.

It didn't really add anything to the conversation, and if I wanted to know what an LLM thought, I'd ask it. The reason for the rule is people come here to interact with other people.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#34

Post by SwellJoe »

dirtikiti wrote: post #5

"Local AI should be a default, not a privilege: private data, no per-token bill, no vendor lock-in. The hardware to run capable models already sits on desks. The software to run those chips well doesn't." So figure out how to run it on Vulkan instead of requiring the user to be locked into expensive CUDA cards.

You can run pretty much every model on Vulkan, including the Qwen MoE models. You can also run pretty much every model on ROCm, Apple Silicon via MLX, and Intel hardware via OpenVINO. Nvidia got there first, but they're no longer clearly dominant in the self-hosting space, simply because of the high cost. I think Apple probably has the lead there, due to unified memory allowing big models to run without multiple big dedicated GPUs, but stuff like Strix Halo with 128GB of unified memory is also pretty much sold out everywhere. There's a lower bound on how small a model can be and still be useful.

Anyway, I don't have any Nvidia hardware, and I've got several local models running and/or training at all times.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#35

Post by CoolGuySteve »

dist-epoch wrote: post #26
doubled112 wrote: post #13

Earlier quoted context omitted.

Depends on your definition of budget friendly, I suppose. I was looking around the other day and the cheapest working 24GB RTX 3090 on eBay was $1800 CAD after exchange rate, shipping and all the rest. Hugely inflated from the $700 they were once going for. Maybe there are still deals around.

Because they are hugely more useful now than running some stupid game at 240 fps instead of 60 fps.

They're not a particularly fast card compared to something like a 5070, they have lots of VRAM.

That's why they were cheap before.

Also "Some stupid game", who woke up and made you king of hobbies.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#36

Post by UncleOxidant »

varispeed wrote: post #24

What is the point of this if such small model generally produces rubbish?

Have you tried out Qwen3.5-27b? It's quite amazing that a model with only 27b parameters can do what it's doing. I've had it working on a project that has C++, python and Verilog code. It's generating code in all 3 and very competently. I've had it look into other git repos to bring in ideas from them into this one. Again, it's doing an amazingly good job with this and it's running locally on my PC.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#37

Post by fluoridation »

doubled112 wrote: post #13
VladVladikoff wrote: post #10

Earlier quoted context omitted.

That's a pretty popular budget friendly GPU people use for local AI, it actually seems like an excellent choice IMHO.

Depends on your definition of budget friendly, I suppose. I was looking around the other day and the cheapest working 24GB RTX 3090 on eBay was $1800 CAD after exchange rate, shipping and all the rest. Hugely inflated from the $700 they were once going for. Maybe there are still deals around.

That's insane. I bought two in December for ARS 1.2M (a little less than USD 1000). Maybe OpenClaw raised the demand.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#38

Post by fluoridation »

tempaccount5050 wrote: post #30
doctorpangloss wrote: post #11

Earlier quoted context omitted.

AI authored comments are against the rules. that said what is the point of engaging here if you won't do it with your own words? like do you understand any of what you wrote?

I find these comments hilarious. Are we supposed to build AI and then not use it? Super goofy.

Not "not use it". Not use it to make people believe they're talking to real people.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#39

Post by VladVladikoff »

doubled112 wrote: post #13
VladVladikoff wrote: post #10

Earlier quoted context omitted.

That's a pretty popular budget friendly GPU people use for local AI, it actually seems like an excellent choice IMHO.

Depends on your definition of budget friendly, I suppose. I was looking around the other day and the cheapest working 24GB RTX 3090 on eBay was $1800 CAD after exchange rate, shipping and all the rest. Hugely inflated from the $700 they were once going for. Maybe there are still deals around.

Wild I paid $1000 CAD for mine 2 years ago, I guess things have changed.

Re: We got 207 tok/s with Qwen3.5-27B on an RTX 3090

#40

Post by Aurornis »

GreenGames wrote: post #27
Aurornis wrote: post #16

This is a Claude-code generated repo that implements some ideas from research papers. If you follow this space, every paper release spawns tens or hundreds of vibecoded repos like this that get spammed to Reddit, Hacker News, and other sites. It's generally best to overlook the vibecoded repos and go closer to the source for up to date information. In this case, z-lab already showed Qwen3.5-27B with DFlash last month…

This reads like you didn’t read the post. z-lab runs BF16 on B200 (54+ GB). There is no z-lab path that fits on a 24 GB 3090. That is literally the entire point of our work, and it is stated in the second paragraph. If you had checked the HF model card you linked before posting, you would see the same thing. Before this repo, there was no path to run this... SGLang's GGUF path for this model is broken. llama.cpp does…

> This reads like you didn’t read the post.

I was discussing details I read in your repo. How did you conclude that I didn't read the post? I'm skeptical a human is writing these comments because everything you're posting reads like LLM output

> On the Q4 KV cache: the tradeoff is disclosed with actual numbers. AL 8.56 -> 8.33 at short context (3% drop), dramatically better at long context.

I'm sorry, but you're not the first (or LLM) to think of using Q4 KV cache to fit more context in VRAM.

The degradation is far more than 3% on real evals. Q8 only recently became usable on Qwen3.5 in llama.cpp with the context rotation changes. Before that bf16 was necessary to get decent performance in real tasks.

Q4 is a non-starter for real work. The fact that you're still trying to defend it tells me you haven't used this for anything other than token/sec racing.

Post reply on HN