Local Qwen isn't a worse Opus, it's a different tool • Hacker News Board

Re: Local Qwen isn't a worse Opus, it's a different tool

#161

Post by scotty79 » Thu, Jun 18, 2026, 12:46 PM UTC

coldtea wrote: post #134

scotty79 wrote: post #126

Earlier quoted context omitted.

If you have a toolbox full of similar but different tool getting to know them is a prudent thing to do, not a psychosis. There's no connection because the tool is immutable (except for adjustments you made) but you do develop a specific relation with that tool. Some people even love some of their tools at some level. And if humans are anything, they are tool users.

> If you have a toolbox full of similar but different tool getting to know them is a prudent thing to do, not a psychosis Can be both. Use of some tools like LLMs might be more inducing psychosis than others like plain compilers or hammers. > And if humans are anything, they are tool users. To the point of self-destruction sometimes.

> Use of some tools like LLMs might be more inducing psychosis than others like plain compilers or hammers.

I really don't get it. Why the fact that it outputs words is so goddamn important for everybody? How does it suddenly make you so emotionally vulnerable? Does my brain work in a different way than the rest of humanity? Can't you disregard what's irrelevant? Is every programmer suddenly a trump supporter that has no ability to recognize empty words? To recognize lies about emotions and facts?

Words are just input. Mostly garbage. Emotion inducing words are garbage 10 times more often than any other. I could expect romance reader to be affected, or somebody with iq 70. But how the caste of some of the most technical people ever is afraid of catching psychosis just because they might read some words?

Re: Local Qwen isn't a worse Opus, it's a different tool

#162

Post by alexellisuk » Thu, Jun 18, 2026, 12:47 PM UTC

zkmon wrote: post #50

The seems to talk a lot about 27B. In my experience, I saw 35B-A3B to be equally good in quality and the MoE gave more tg/s.

The important thing about MoEs which I mention in the conclusion is that they carry fewer (way fewer) active tokens during inference/generation.

35B-A3B is what we started out with in the days of only having the 3090, but the quality is not as good, and the speed from the cards we have now can blaze at 130-200 tokens per second of generation with q5 and a full context in fp16.

Not to say that MoEs don't have their place. For people running on unified RAM, they're sometimes the only viable option due to the slowness of dense models.

Why is a dense model slower? All model weights have to be loaded and exercised. Passing through 27B vs 3B (active) is maths. So yes you will always get more tokens per second of generation.

You must (just as we did) evaluate on your own products and daily work. If the MoE gives the results you need with only 3B parameters then you have your answer.

Not prescriptive at all. This is experience based, from the trenches of a actual software business so hopefully a different perspective for folks than "Ran Qwen on my macbook, generated a great python script for me"

Re: Local Qwen isn't a worse Opus, it's a different tool

#163

Post by stingraycharles » Thu, Jun 18, 2026, 12:48 PM UTC

willtemperley wrote: post #38

stingraycharles wrote: post #10

Earlier quoted context omitted.

I agree with your general gist, and in general it’s a “the best tool for the particular job”, keeping token spent and other things in mind as well. What I do know absolutely for sure is that LLM benchmarks are not to be trusted, they are just a minor indicator and real world usage is often very different.

Yes, how do we know Opus 4.8 hasn't been trained on the SWE-Bench examples? With a squillion dollars at stake per bench point, someone will have figured out a plausibly deniable way to game these benchmarks.

Ehr, the SWE bench examples are particularly horrible as those are just publicly available historical PRs. So if the models are trained on GitHub data, it will be included.

So almost by design that particular benchmark is tainted, and benchmarks recall rather than reasoning.

Re: Local Qwen isn't a worse Opus, it's a different tool

#164

Post by alexellisuk » Thu, Jun 18, 2026, 12:50 PM UTC

watt wrote: post #98

I find it strange that software people will accept this level of flakiness from the hardware. Normally you would just send the card back, and request a replacement. > One of the cards would only show up if I crossed my fingers when turning it on. Even reboots wouldn't cure it - I had to A/C power off and remove the power cable each time for 30 seconds. This is ridiculous. Of course we are living through supply crunch…

Ha, you underestimate how dogged you need to be to get this stuff working well.

The RTX 3090 in question was used from eBay, no way to return it. The RTX 6000 Pro is the "new card" in question here. The 3090s remain an interesting playground for testing things like VFIO passthrough for SlicerVM and other models whilst not interrupting people on the newer card.

In the end, the most stable fix I've found is to install the older proprietary driver and disable the GSP firmware. Have had no issues since.

So "clearly defective hardware" seems like it may not be quite correct. And the thing that kept me coming back - along with not having a suitable replacement - or having to gamble on eBay again was the reliability once it showed up in nvidia-smi.

Re: Local Qwen isn't a worse Opus, it's a different tool

#165

Post by alexellisuk » Thu, Jun 18, 2026, 12:55 PM UTC

krzyk wrote: post #116

3090 and 2x3090 are quite popular. But if you uses gigantic (for local models) context of 200k it will go south pretty quickly - any quantization of context quickly becomes the issue.

I think that's quite telling Gorgi replied that he uses Qwen with 131k context.

https://x.com/ggerganov/status/2067539416436867230?s=20

We also use it with 200-256k (native) context length.

The issue could be that folks that don't see looping aren't pushing the model as hard, or as enthusiastically.

We also had far fewer issues when thinking was turned off, than with a reasoning budget capped at 2048.

Some fine-tunes like Qwopus-Coder just seem prone to looping - google it, you'll see plenty of reports, even on Reddit.

For what it's worth seen the RTX 6000 Pro loop even at fp16 on the KV cache - and with vLLM.

Re: Local Qwen isn't a worse Opus, it's a different tool

#166

Post by hootz » Thu, Jun 18, 2026, 12:59 PM UTC

stego-tech wrote: post #131

I still believe that the strength of AI is when it can be applied locally in a secure and private manner, rather than yet another cloud-based service you must pay for indefinitely even as it gets progressively worse to satiate the greed of corporate shareholders. ChatGPT and Anthropic will never, ever get me to tie my Health Data to their systems, but I still believe in the capabilities of AI in identifying patterns…

The thing about "local" models for me is that they usually mean open-weight (and maybe open-source too), so they can be used locally, yes, but they can also be hosted by independent providers! With models like Qwen, DeepSeek and others, you aren't tied to a single corp, you can switch between indie providers, some of which may give you better privacy guarantees. That allows you to use the models even on devices uncapable of running them, if they have an Internet connection.

The strength with AI is with open-source models. We need to keep away from vendor lock-in and use models that allow both local usage and hosting by independent providers.

Re: Local Qwen isn't a worse Opus, it's a different tool

#167

Post by Wowfunhappy » Thu, Jun 18, 2026, 1:00 PM UTC

sanderjd wrote: post #14

stingraycharles wrote: post #10

Earlier quoted context omitted.

I agree with your general gist, and in general it’s a “the best tool for the particular job”, keeping token spent and other things in mind as well. What I do know absolutely for sure is that LLM benchmarks are not to be trusted, they are just a minor indicator and real world usage is often very different.

I share this sense, but my immediate thought is that we need to improve the evaluations! Do you think this is impossible? That there is something indelible that it is not possible to capture empirically? I kind of have this intuitive sense that it is this way, but simultaneously I think that it's unlikely to really be true.

I'm not GP, but yes, I think it's impossible.

Take AI out of the picture for a moment. What makes someone a good coder? What makes someone intelligent? How do you evaluate those skills?

Of course we have standardized tests, and they're useful, but they're also imperfect. And they become especially imperfect when people start training for the tests specifically—which is, essentially, benchmaxxing.

We have never been able to quantitatively measure most skills to a high degree of accuracy, despite centuries of trying. That's not going to change now.

(I don't mean to anthropomorphize the LLMs, but I do think they're like humans in this way.)

Re: Local Qwen isn't a worse Opus, it's a different tool

#168

Post by HappySweeney » Thu, Jun 18, 2026, 1:04 PM UTC

renegade-otter wrote: post #157

ACCount37 wrote: post #111

Earlier quoted context omitted.

One issue with that is that human helpers last longer. LLMs cycle in and out in months, and what held for Your Favorite LLM 6.7 may not hold for Your Favorite LLM 6.9.

Right, this is why I would slam the breaks on investing into your workflow all of your time and effort, because 2 months from now it may be out the window. Frontier models are also constantly being tweaked, so what worked yesterday may be off today. ChatGPT was obedient with the grill-me technique, just wrote a plan. Yesterday it started jumping to implementation. Why?

I find that when an LLM jumps into tasks it was not told to do (or even worse, doing things it was explicitly told not to), it is a good sign the context is too full, and you should do a controlled hand-off to a new instance.

Re: Local Qwen isn't a worse Opus, it's a different tool

#169

Post by piterrro » Thu, Jun 18, 2026, 1:05 PM UTC

This is amazing but for everyone out there wanting to buy and build your own AI rig I recommend connecting to one of mamy inference providers and trying out different models themselves for a while. Costs pennies but can give you a nice preview of what you can get with your own rig. Just a friendly tip.

Re: Local Qwen isn't a worse Opus, it's a different tool

#170

Post by navbaker » Thu, Jun 18, 2026, 1:06 PM UTC

barrkel wrote: post #67

I found it interesting that vLLM was dismissed as slower than llama.cpp. IME vLLM is quite a bit faster than llama.cpp but where it really wipes the floor with it is in batching concurrent load. The downside is that it is dramatically less flexible in terms of tweaking. It gives you very few options for running quantized weights. It takes a lot longer to start up because it optimizes the compute graph. So for single…

Yeah, I was a bit baffled by the author complaining about cache prefixes getting destroyed when more than one user hit the model, but then continuing to use llama.cpp instead of switching to vLLM.