Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

imil.net

Post reply on HN
Print view View original

91–100 of 116 posts

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#91

Post by epolanski »

iamanllm wrote: post #90
sieste wrote: post #11

That's almost exactly my setup and I'm very happy with its performance. I noticed recently that I started to prefer my local Qwen3.6 35B A3B and pi agent over Claude Code. Both fail at different tasks, and Qwen more so than Claude. But the way Qwen fails is much more straightforward. In writing tasks Qwens hallucinations and bullshitting are much easier to spot because it doesn't have the sleek vocabulary and wordsmi…

Frontier models are still better (everyone would use them if it was cheap). Open source models are capable on even non "simple" problems but I trust them less, even though I usually write plans for all changes, and they are worse at debugging. I recently converted my homelab to nixos and let's just say Deepseek failed and Fable did great (the night before getting killed)

While what you say is in general true, every model that followed Opus 4.6 on Anthropic side has been increasingly worse at what the previous user points out: they are extremely smart and can convince the user about major falsehood.

They are way too trained/reinforced on solving problems rather than assisting you, something on which they have becoming extremely bad at.

It's hard to explain because I too had the many moments where "Fable5 / Opus4.8 xhigh could solve bugs/stuff that previous models couldn't", I know that to be true, and they are very useful for that.

But 90% of my tasks are quite mundane and I need thorough investigation and a proper assistant. Not a smart bullshitter fixated on solving the issue itself. On that Opus 4.6 has been the last good model.

Anything after that is completely skewed towards passing benchmarks and E2E tasks, but definitely not assisting.

Fable in particular was a disaster on that, non stop being thorough on the fix it fixated on, writing nthousand experiments in /tmp, etc. Great model, not gonna lie, but only if your focus is vibe coding and you accept that you're nothing but an assistant and accept its shortcomings.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#92

Post by ericb »

sieste wrote: post #11

That's almost exactly my setup and I'm very happy with its performance. I noticed recently that I started to prefer my local Qwen3.6 35B A3B and pi agent over Claude Code. Both fail at different tasks, and Qwen more so than Claude. But the way Qwen fails is much more straightforward. In writing tasks Qwens hallucinations and bullshitting are much easier to spot because it doesn't have the sleek vocabulary and wordsmi…

Do the two cards "share" their memory pool? Can work still be split across it? I'm wondering how it would do with something like fine tuning?

Do you get the speed of the 5080 with the memory of the 3090?

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#93

Post by aand16 »

DiabloD3 wrote: post #51

The recommended values for Qwen 3.6 in thinking mode is `--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00`, and `--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00` for coding/tool calling tasks, and for non-thinking, `--temp 0.7 -top-p 0.8 --top-k 20 --presence-penalty 1.5 --min-p 0.00`. The options listed are none of these. Also, the recommended Qwen MTP settings are `--spec-type draft-mtp --spec-draft-n-max 2`. 3 is n…

> You can also add `ngram-mod`, but after `draft-mtp`

It looks like there's a hardcoded preference, CLI order is not important.

(speculative.cpp:1322-1381): common_get_enabled_speculative_configs converts the types vector to a bitmask (order-independent). Then configs are added in a hardcoded priority order:

ngram-simple

ngram-map-k

ngram-map-k4v

ngram-mod

ngram-cache

draft-simple

draft-eagle3

draft-mtp

(speculative.cpp:1557-1603): common_speculative_draft iterates impls in the hardcoded priority order. Once an impl produces a draft for a sequence, later impls skip that sequence.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#94

Post by iamanllm »

epolanski wrote: post #91
iamanllm wrote: post #90

Earlier quoted context omitted.

Frontier models are still better (everyone would use them if it was cheap). Open source models are capable on even non "simple" problems but I trust them less, even though I usually write plans for all changes, and they are worse at debugging. I recently converted my homelab to nixos and let's just say Deepseek failed and Fable did great (the night before getting killed)

While what you say is in general true, every model that followed Opus 4.6 on Anthropic side has been increasingly worse at what the previous user points out: they are extremely smart and can convince the user about major falsehood. They are way too trained/reinforced on solving problems rather than assisting you, something on which they have becoming extremely bad at. It's hard to explain because I too had the many m…

yeah, the "proactivity" of recent anthropic models and sophisticated bullshitting are bad, although my experience is that even on simple tasks i've never used a oss model that has consistently been better in terms of the quality of the result.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#96

Post by hnthrow0287345 »

sieste wrote: post #11

That's almost exactly my setup and I'm very happy with its performance. I noticed recently that I started to prefer my local Qwen3.6 35B A3B and pi agent over Claude Code. Both fail at different tasks, and Qwen more so than Claude. But the way Qwen fails is much more straightforward. In writing tasks Qwens hallucinations and bullshitting are much easier to spot because it doesn't have the sleek vocabulary and wordsmi…

>In writing tasks Qwens hallucinations and bullshitting are much easier to spot because it doesn't have the sleek vocabulary and wordsmithing skills to disguise its ignorance.

Can't wait until we just remove the language from the LLMs for accuracy and efficiency

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#97

Post by otabdeveloper4 »

redox99 wrote: post #88
nullbio wrote: post #68

Earlier quoted context omitted.

I know the big labs like to pretend that their models are trillion parameter. But how likely is that really to be the case when Qwen 3.6 35B A3B gets so close to their performance? Seems that with the best research applied, best training data, they'd be able to top the charts with a 60B model quite easily.

Qwen 35B isn't even remotely close to the big models. It's just people over hyping small models. Ignore the benchmarks they are almost meaningless. If you want something comparable you need the trillion parameter open models like deepseek.

Number of parameters doesn't make the model smarter, it just makes it know more stuff out of the box.

At some point there's diminishing returns and your coding LLM performs worse because you encoded useless stuff like Pokemon combinations or languages you don't speak into its parameter space.

The "smartness" of the model comes from RLHF post-training, which is orthogonal to model size.

Also, if you're using an agentic harness a much better approach is to let the model control its own context. If you ever reach a point where your coding LLM needs to know about Pokemon, just give it a web search tool and let it google the Pokemons.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#98

Post by [deleted] »

PeterStuer wrote: post #27
deng wrote: post #4

I can understand the joy of running things yourself, and can also see the privacy aspect. However, I pay ~3$ per 1M/tokens for that model on Openrouter, and it's not even quantized. A refurbished 3090 and a 5080 will set you back well over 2k, not to mention the electricity to run them...

When they declare open models a 'security risk', his setup will be running, yours will not and even that 3090 will be way outside of your reach.

[deleted]

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#99

Post by redox99 »

otabdeveloper4 wrote: post #97
redox99 wrote: post #88

Earlier quoted context omitted.

Qwen 35B isn't even remotely close to the big models. It's just people over hyping small models. Ignore the benchmarks they are almost meaningless. If you want something comparable you need the trillion parameter open models like deepseek.

Number of parameters doesn't make the model smarter, it just makes it know more stuff out of the box. At some point there's diminishing returns and your coding LLM performs worse because you encoded useless stuff like Pokemon combinations or languages you don't speak into its parameter space. The "smartness" of the model comes from RLHF post-training, which is orthogonal to model size. Also, if you're using an agenti…

That's just... not true. Just compare any open model which is trained with the same recipe but multiple sizes.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#100

Post by arjie »

ydj wrote: post #81
arjie wrote: post #31

Earlier quoted context omitted.

That’s the cost of using a new hardware provider. A single RTX Pro 6000 Blackwell Max-Q will do better than that and be much more usable. I have 2 running DS4 Flash at 160 tok/s with max num seqs 4. Very interesting though, these Tenstorrent chips. Might get one to experiment with.

Yeah that’s definitely the smarter buy if you want to just have models running quickly. But the cost of 2 p150 and a 4090 was The main issue is the immature software, and somewhat baroque way of writing kernels. Please, buy one and join us.

Were you able to connect the two P150 using the qsfp-dd cable? They only sell 4x and 8x topologies so I’m curious if that worked for you. Are you able to run them tensor parallel?
Post reply on HN