Local Qwen isn't a worse Opus, it's a different tool • Hacker News Board

Re: Local Qwen isn't a worse Opus, it's a different tool

#211

Post by alexellisuk » Thu, Jun 18, 2026, 3:53 PM UTC

skipants wrote: post #179

I feel like it's the Emperor's new clothes reading this article and seeing the praise it's getting. This sentence doesn't even make sense: > These products use very low level Linux primitives like containers, Kubernetes, Firecracker microVMs, and networked protocols. Out of anything that is a "low level linux primitive" I could maybe argue that networking? protocols fit the bill. And it's obviously fully AI-generated…

Fair enough, that sentence was fairly compressed. I’ve reworded it - the meaning remains the same.

The post is not AI generated, I use AI for code generation and write my own articles.

Which part of the post are you struggling with? This is a post describing our own experience and journey. Happy to back up any specific claim.

Re: Local Qwen isn't a worse Opus, it's a different tool

#212

Post by alexellisuk » Thu, Jun 18, 2026, 3:59 PM UTC

ttsiodras wrote: post #207

Interesting article. IMHO, the author could have done two things better: - vllm instead of llama.cpp. With NVIDIA HW, there is huge difference in multi-user loads and caching with vllm; when he was complaining about what happens when more than one user uses the model, and about losing caching, I was "well, duh". - The budget he used for a single card could have instead be put to far, far better use with SPARKs. I hav…

We did run vLLM on the 3090s — measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning.

It's the right call for concurrent batched serving (barrkel's point downthread is spot on), but for how we use it llama.cpp is still better for us.

The Spark/GX10 route is a genuinely different bet though and appreciate you sharing your numbers. At the time (several months ago) the consensus was that GX10s were for fine-tuning only, and the numbers were severely low.

..and the card was never about replacing a Claude Max sub. For the workloads we actually bought it for, it's giving us 140-200 tok/s (which matters).

Re: Local Qwen isn't a worse Opus, it's a different tool

#213

Post by themgt » Thu, Jun 18, 2026, 4:00 PM UTC

mahidhar wrote: post #173

coldtea wrote: post #103

Earlier quoted context omitted.

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

Well, unlike a human, I cannot expect any these LLMs to take any ownership of the work they do. I cannot expect any given model and version (sonnet 4.6) to learn, improve and adapt over time. I cannot expect it's limitations to ever go away at the model level. So it is not like a human in most ways that I actually care about. That said, I can't wait for LLMs to stop being AI and start being just another tool. Anythin…

That said, I can't wait for LLMs to stop being AI and start being just another tool.

From a horse's perspective, the internal combustion engine is just another tool for making scary noises and powering horse trailers to take me on fun horse adventures. So ... perhaps.

Re: Local Qwen isn't a worse Opus, it's a different tool

#214

Post by ttsiodras » Thu, Jun 18, 2026, 4:05 PM UTC

alexellisuk wrote: post #212

ttsiodras wrote: post #207

Interesting article. IMHO, the author could have done two things better: - vllm instead of llama.cpp. With NVIDIA HW, there is huge difference in multi-user loads and caching with vllm; when he was complaining about what happens when more than one user uses the model, and about losing caching, I was "well, duh". - The budget he used for a single card could have instead be put to far, far better use with SPARKs. I hav…

We did run vLLM on the 3090s — measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning. It's the right call for concurrent batched serving (barrkel's point downthread is sp…

I hear you on the insane amount of time vllm takes to launch (atlas is a move in the right direction in that regard).

But mostly I wanted to raise awareness to readers of your article that no, if you want to do inference, paying 15K for a single 96GB card almost certainly makes no sense. Buy 4 GX10s with the same money, and enjoy dramatically better models and user scalability.

Regardless - thanks for putting the effort to share your findings! I keep postponing doing the same... there's tons of things everyone is re-discovering on their own.

Re: Local Qwen isn't a worse Opus, it's a different tool

#215

Post by alexellisuk » Thu, Jun 18, 2026, 4:06 PM UTC

barrkel wrote: post #67

I found it interesting that vLLM was dismissed as slower than llama.cpp. IME vLLM is quite a bit faster than llama.cpp but where it really wipes the floor with it is in batching concurrent load. The downside is that it is dramatically less flexible in terms of tweaking. It gives you very few options for running quantized weights. It takes a lot longer to start up because it optimizes the compute graph. So for single…

vLLM is great at continuous batching and model serving in production, but it's a very different beast and much less versatile for the prosumer category (where we sit for our usage)

Dismissed is a strong term, but let me give you some more details.

It took a good 4 minutes plus to load up on the 2x 3090 rig, and served a single request 3 tokens/second slower.

And the worst bit? With all that work - setting it up and tuning it - it still looped. I was hoping "use just vLLM" advice that we get touted everywhere was the silver bullet.

The only thing I'd caution here is that we don't start bashing on llama.cpp like people did with Ollama. It's a very capable tool and for the use-cases we actually want the card for makes more sense.

For a large team replacing their Claude Subs perhaps vLLM is the only option, but you really need to add about 5 more RTX 6000 cards into the mix, so you can load something like GLM 5.2.

Re: Local Qwen isn't a worse Opus, it's a different tool

#216

Post by alexellisuk » Thu, Jun 18, 2026, 4:12 PM UTC

bee_rider wrote: post #185

Tangential question (since they brought it up in the article) from someone not involved in AI performance optimization: How big of a deal is looping, practically? Or, I mean, I see thinking models loop occasionally. But it seems to me that every token in the loop should be in the KV cache already, is there really no way to either power through a loop because of the 100% cache hit rate, or identify that you are in a l…

1. On the technical:

The cache only makes generation fast, it doesn't influence what gets chosen next. The loops that hurt the most (point 2 below) are when the model re-decides to do the same thing in different words, which is much harder to detect automatically. We're experimenting with repetition penalty and turning thinking off to solve for the 1st kind of looping (below)

2. On "why is looping a problem" for us

Practical example, which I covered in the post: "add --json to every command that does a get or list in faas-cli" - this was a small-ish, open source CLI written with Cobra a very common framework.

If I send that to Claude (any of their models) or Codex (GPT), I would have a fully working solution the next time I opened that terminal - a few seconds - a few minutes.

With the local model, when it loops, you get some progress and start working on something else. Come back, maybe even 30 minutes later and see it's been printing the same 5 lines over and over constantly.

Trust is important for a tool like this, that eroded it.

The other type of loop I mention in the blog post is "unable to solve it" loop - Han ran into that more.

"Oh I need to fix the indent from 8 to 5 characters in main.py" "Wait I don't know how to write Python code" "Oh now it's broken and I don't know what to do, maybe I should stop" "Let me edit ... " etc, etc

Re: Local Qwen isn't a worse Opus, it's a different tool

#217

Post by TeMPOraL » Thu, Jun 18, 2026, 4:20 PM UTC

gib444 wrote: post #121

coldtea wrote: post #103

Earlier quoted context omitted.

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

No, I won't anthropomorphise LLMs.

That's your prerogative, but be aware you'll continue to remain confused about LLMs. Anthropomorphizing them is what gives you the best high-level intuition about where and how to employ them, and where and how not to.

Re: Local Qwen isn't a worse Opus, it's a different tool

#218

Post by sixothree » Thu, Jun 18, 2026, 4:21 PM UTC

bluegatty wrote: post #106

couscouspie wrote: post #93

Earlier quoted context omitted.

That would be ideal, but AI is less like a tool and more like a human in this regard and you don't have character sheets for each of your colleagues, as well.

These are $1 Trillion dollar companies that can't produce explicit details on how their products work? It's nonsense.

I think if they could explain how they work, their strengths and weaknesses, they would reveal to the world whose data they've been appropriating.

Re: Local Qwen isn't a worse Opus, it's a different tool

#219

Post by bluegatty » Thu, Jun 18, 2026, 4:24 PM UTC

sixothree wrote: post #218

bluegatty wrote: post #106

Earlier quoted context omitted.

These are $1 Trillion dollar companies that can't produce explicit details on how their products work? It's nonsense.

I think if they could explain how they work, their strengths and weaknesses, they would reveal to the world whose data they've been appropriating.

That's another thing altogether. They can characterize the behaviour without quite giving up who and where the data comes from.

Admittedly, yes, there's some overlap there.

They would have to admit 'seen it in the training data' as a factor, and that opens a can of worms.

Re: Local Qwen isn't a worse Opus, it's a different tool

#220

Post by devin » Thu, Jun 18, 2026, 4:34 PM UTC

glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

It is not at all like playing an instrument.

Instruments present a clear interface to a user, have predictable outputs, etc.

The only comparison that might work for me is that LLMs are very bad instruments where you are constantly forced to negotiate its idiosyncrasies in order to massage the output you want from it, and even then there is enough randomness that trying to do so is almost a fool's errand.