Local Qwen isn't a worse Opus, it's a different tool • Hacker News Board

Re: Local Qwen isn't a worse Opus, it's a different tool

#11

Post by hbbio » Thu, Jun 18, 2026, 5:04 AM UTC

cptskippy wrote: post #4

I've been running qwen3-5-9b-q4-k-m and qwen3-6-27b-q6-k simultaneously on an Intel Arc Pro B70 with a lot of success. https://github.com/cptskippy/battlemage-llm-gateway Opencode has been a huge productivity accelerator. I have two Hermes agents that I'm training to support my workflow with pretty good success. One is a personal assistant who manages my backlog and keeps me on task, follows up with me on items, and…

Interesting setup, thx for sharing.

How many tokens/sec do you get with 27b? Are you using MTP?

Re: Local Qwen isn't a worse Opus, it's a different tool

#12

Post by appplication » Thu, Jun 18, 2026, 5:05 AM UTC

zmmmmm wrote: post #9

That's a great write up. The one thing I feel it seems to under estimate is the likelihood of improvement. Even the authors acknowledge it's not even worth comparing local models from a year ago to what we have now. In fact, people widely see Opus 4.5 in November last year - 8 months ago - as the first time agentic coding became viable broadly viable even with frontier hosted models. So why would we lock in hard on a…

Agree 100%, even on claude 4.5 being the turning point for agentic coding. It completely turned me around on it.

Re: Local Qwen isn't a worse Opus, it's a different tool

#13

Post by rippeltippel » Thu, Jun 18, 2026, 5:11 AM UTC

zmmmmm wrote: post #9

That's a great write up. The one thing I feel it seems to under estimate is the likelihood of improvement. Even the authors acknowledge it's not even worth comparing local models from a year ago to what we have now. In fact, people widely see Opus 4.5 in November last year - 8 months ago - as the first time agentic coding became viable broadly viable even with frontier hosted models. So why would we lock in hard on a…

Since the author is referring to a specific model, I think it makes sense to ignore how the model (or local models in general) may improve over time.

It's like buying a car: I drive that car and get attuned to its characteristics; I don't think how that car (or similar cars) may improve. That's my tool and I want to make the most of it.

It is true that switching a local models it technically very cheap, but there's a considerable time investment in squeezing the most out of it, which may not work on a newer version of that model.

Re: Local Qwen isn't a worse Opus, it's a different tool

#14

Post by sanderjd » Thu, Jun 18, 2026, 5:12 AM UTC

stingraycharles wrote: post #10

glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

I agree with your general gist, and in general it’s a “the best tool for the particular job”, keeping token spent and other things in mind as well. What I do know absolutely for sure is that LLM benchmarks are not to be trusted, they are just a minor indicator and real world usage is often very different.

I share this sense, but my immediate thought is that we need to improve the evaluations! Do you think this is impossible? That there is something indelible that it is not possible to capture empirically? I kind of have this intuitive sense that it is this way, but simultaneously I think that it's unlikely to really be true.

Re: Local Qwen isn't a worse Opus, it's a different tool

#15

Post by sanderjd » Thu, Jun 18, 2026, 5:14 AM UTC

zmmmmm wrote: post #9

That's a great write up. The one thing I feel it seems to under estimate is the likelihood of improvement. Even the authors acknowledge it's not even worth comparing local models from a year ago to what we have now. In fact, people widely see Opus 4.5 in November last year - 8 months ago - as the first time agentic coding became viable broadly viable even with frontier hosted models. So why would we lock in hard on a…

Right. Opus 4.5 8 months ago, good enough for agentic coding. How far behind that are open weight models? More than 8 months? But how much more? When will they reach Opus 4.5 level? A few months from now? A year from now? Never?

Re: Local Qwen isn't a worse Opus, it's a different tool

#16

Post by sanderjd » Thu, Jun 18, 2026, 5:16 AM UTC

gpt5 wrote: post #2

This article is a good summary of local models. Unlike the way they are hyped sometimes, as fantastic tools for coding and agentic local work. The reality is that they are rather limited, would not do well on a long or complex task, and are prone to fall into loops, forget their tasks, etc. Not mentioned in the article is that they are also rather expensive - not just for the hardware cost, but also electricity. Thes…

But that's current hardware. What about future hardware? What about hardware optimized for inference? What about hardware optimized to run a particular model?

Re: Local Qwen isn't a worse Opus, it's a different tool

#17

Post by jauntywundrkind » Thu, Jun 18, 2026, 5:16 AM UTC

cptskippy wrote: post #4

I've been running qwen3-5-9b-q4-k-m and qwen3-6-27b-q6-k simultaneously on an Intel Arc Pro B70 with a lot of success. https://github.com/cptskippy/battlemage-llm-gateway Opencode has been a huge productivity accelerator. I have two Hermes agents that I'm training to support my workflow with pretty good success. One is a personal assistant who manages my backlog and keeps me on task, follows up with me on items, and…

What's the value running the smaller model too? Why not just the big model for everything? I note both are dense, as well.

Re: Local Qwen isn't a worse Opus, it's a different tool

#18

Post by vkazanov » Thu, Jun 18, 2026, 5:23 AM UTC

glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

The problem is not that there details, the problem is constantly shifting ground. We can only rlpy on a harness to be sort of predictable but the models change all the time.

Re: Local Qwen isn't a worse Opus, it's a different tool

#19

Post by theplumber » Thu, Jun 18, 2026, 5:32 AM UTC

sanderjd wrote: post #15

zmmmmm wrote: post #9

That's a great write up. The one thing I feel it seems to under estimate is the likelihood of improvement. Even the authors acknowledge it's not even worth comparing local models from a year ago to what we have now. In fact, people widely see Opus 4.5 in November last year - 8 months ago - as the first time agentic coding became viable broadly viable even with frontier hosted models. So why would we lock in hard on a…

Right. Opus 4.5 8 months ago, good enough for agentic coding. How far behind that are open weight models? More than 8 months? But how much more? When will they reach Opus 4.5 level? A few months from now? A year from now? Never?

I think in the next 6 months we will have Opus 4.5 performance in open models. We are very close

Re: Local Qwen isn't a worse Opus, it's a different tool

#20

Post by Ritewut » Thu, Jun 18, 2026, 5:34 AM UTC

jauntywundrkind wrote: post #17

cptskippy wrote: post #4

I've been running qwen3-5-9b-q4-k-m and qwen3-6-27b-q6-k simultaneously on an Intel Arc Pro B70 with a lot of success. https://github.com/cptskippy/battlemage-llm-gateway Opencode has been a huge productivity accelerator. I have two Hermes agents that I'm training to support my workflow with pretty good success. One is a personal assistant who manages my backlog and keeps me on task, follows up with me on items, and…

What's the value running the smaller model too? Why not just the big model for everything? I note both are dense, as well.

Tokens per second. The difference between 8B and something like 16B is not as big as you might think in practical usage and 8B is a lot faster and interactive than 16B but there are certain things where it is useful to farm it out to the large model.