Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

imil.net

Post reply on HN
Print view View original

61–70 of 116 posts

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#61

Post by matthewfcarlson »

eurekin wrote: post #28
sieste wrote: post #11

That's almost exactly my setup and I'm very happy with its performance. I noticed recently that I started to prefer my local Qwen3.6 35B A3B and pi agent over Claude Code. Both fail at different tasks, and Qwen more so than Claude. But the way Qwen fails is much more straightforward. In writing tasks Qwens hallucinations and bullshitting are much easier to spot because it doesn't have the sleek vocabulary and wordsmi…

I keep finding more and more usecases for Q3.6 27b (same league) and the best performance is, when answers to my question is already in the context. The moment I'm trying something open-ended or ambitious, Claude/ChatGPT clearly take you to the goal quicker. For things, where there's a way to build a knowledgebase though, the local llm definitely can be a true contender. Plus, having a big context and no worries abou…

I keep playing around with this exact concept. While I don’t always trust entirely AI generated recipe, more traditional setups are super rigid when it comes to ingredients

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#62

Post by mappu »

alexjplant wrote: post #38
deng wrote: post #4

I can understand the joy of running things yourself, and can also see the privacy aspect. However, I pay ~3$ per 1M/tokens for that model on Openrouter, and it's not even quantized. A refurbished 3090 and a 5080 will set you back well over 2k, not to mention the electricity to run them...

I've spent the past week trying to scheme a way to get affordable local inference of something useful (Qwen3.6-36B-A3B) for ~$500 and have come to the conclusion that it simply isn't viable. A pair of power-restricted P100s in a workstation gets close but the workstations themselves are expensive and rare as hen's teeth (not to mention loud and large). I think early '27 will be when things open up as the hardware mar…

I'm running Qwen3.6-35B-A3B on a very ordinary desktop PC (32GB DDR5, 8GB Radeon 6600XT) and getting a useful 15-20 tok/sec out of it. The MoE architecture and auto offloading from system to VRAM is just fantastic. Unsloth Q4_K_XL.

The Qwen3.6-27B is unbearably slow as it doesn't fit in VRAM, though, i think the MoE is very easy to run.

It is also extremely nice that you can just `apt install llama.cpp libggml0-backend-vulkan` now too.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#63

Post by trueno »

sieste wrote: post #11

That's almost exactly my setup and I'm very happy with its performance. I noticed recently that I started to prefer my local Qwen3.6 35B A3B and pi agent over Claude Code. Both fail at different tasks, and Qwen more so than Claude. But the way Qwen fails is much more straightforward. In writing tasks Qwens hallucinations and bullshitting are much easier to spot because it doesn't have the sleek vocabulary and wordsmi…

i keep seeing people talk about pi harnesses. whats this about?

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#65

Post by irishcoffee »

It is absolutely mind blowing to see some of the responses here. Open source, run-your-own, pay for nothing, we’re-all-nerds-that-buy-the-hardware-anyways ethos seems basically dead.

I guess I’m getting old. I own two 16gb cards and I use them for models, for gpu-pasthru for gaming, 3d model rendering, etc. 14 year old me is mortified at this community.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#66

Post by z3t4 »

hamburglar wrote: post #55
sieste wrote: post #11

That's almost exactly my setup and I'm very happy with its performance. I noticed recently that I started to prefer my local Qwen3.6 35B A3B and pi agent over Claude Code. Both fail at different tasks, and Qwen more so than Claude. But the way Qwen fails is much more straightforward. In writing tasks Qwens hallucinations and bullshitting are much easier to spot because it doesn't have the sleek vocabulary and wordsmi…

Not having a lot of experience with this, I ask a naive question: is there a world where you can take your local LLM and hook it up to Claude and get more Claude-like results from your local model? Obviously, there are going to be material differences in how these perform, but are we getting close to a place where this is viable? I imagine that the answers are a combination of “not yet” and “yes but it’s a lot slower…

opencode is like Claude code, but you can use any model.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#68

Post by nullbio »

sieste wrote: post #11

That's almost exactly my setup and I'm very happy with its performance. I noticed recently that I started to prefer my local Qwen3.6 35B A3B and pi agent over Claude Code. Both fail at different tasks, and Qwen more so than Claude. But the way Qwen fails is much more straightforward. In writing tasks Qwens hallucinations and bullshitting are much easier to spot because it doesn't have the sleek vocabulary and wordsmi…

I know the big labs like to pretend that their models are trillion parameter. But how likely is that really to be the case when Qwen 3.6 35B A3B gets so close to their performance? Seems that with the best research applied, best training data, they'd be able to top the charts with a 60B model quite easily.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#69

Post by MisterKent »

nullbio wrote: post #68
sieste wrote: post #11

That's almost exactly my setup and I'm very happy with its performance. I noticed recently that I started to prefer my local Qwen3.6 35B A3B and pi agent over Claude Code. Both fail at different tasks, and Qwen more so than Claude. But the way Qwen fails is much more straightforward. In writing tasks Qwens hallucinations and bullshitting are much easier to spot because it doesn't have the sleek vocabulary and wordsmi…

I know the big labs like to pretend that their models are trillion parameter. But how likely is that really to be the case when Qwen 3.6 35B A3B gets so close to their performance? Seems that with the best research applied, best training data, they'd be able to top the charts with a 60B model quite easily.

They want people to believe they have massive models, that is effectively their moat at this point.

Because if they don't imply that size is needed for every task, they'll end up tanking their valuations.

https://blog.nilesh.io/post/ai-profit-race

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#70

Post by nullbio »

irishcoffee wrote: post #65

It is absolutely mind blowing to see some of the responses here. Open source, run-your-own, pay for nothing, we’re-all-nerds-that-buy-the-hardware-anyways ethos seems basically dead. I guess I’m getting old. I own two 16gb cards and I use them for models, for gpu-pasthru for gaming, 3d model rendering, etc. 14 year old me is mortified at this community.

Times are changing. The open-weight models have needed time to catch up, but they're finally at a point now where we can get almost frontier level capabilities for coding.

I just wish we had a way to actually benchmark them properly though. Still seems no one has solved the problem of software architecture, brittleness and bloat as the codebase grows. Models love to add stuff, but they rarely clean up as they go. In a perfect world they'd do both near equally as they're developing.

It would be nice if there was an "architecture quality" benchmark that distilled the essence of what it means to have a good architecture, but I suppose that's an open research question with a lot of variables? Like how is good architecture actually quantified and measured? Is there a mechanism that can be re-used across all codebases to clearly denote one that is good and one that is bad, or is it highly subjective and depend on the lens you're looking at it from? Is there a lot more to it than just "how much refactoring effort is required to extend this in the future?".

Surely this is something that has been well researched - yet I never really hear anything about it. Makes me wonder why.

Post reply on HN