Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local Qwen isn't a worse Opus, it's a different tool

blog.alexellis.io

Post reply on HN
Print view View original

81–90 of 271 posts

Re: Local Qwen isn't a worse Opus, it's a different tool

#81

Post by teh »

I sometimes wonder how much of intelligence is being good with tools.

I feel pretty averagely smart but give me some good tooling like a good editor, a good type system, semantic grep, good testing and some solvers and I can actually deliver some work.

Maybe the trick isn't 500 billion parameters but a model super integrated with the task at hand for iteration and debugging?

FWIW the article really mirrors my own experience. I can run a small gemma4 for quick edits (and it's fast!) or data cleanup but for other tasks you do need a different tool (claude).

Re: Local Qwen isn't a worse Opus, it's a different tool

#82

Post by walthamstow »

theshrike79 wrote: post #46
gpt5 wrote: post #2

This article is a good summary of local models. Unlike the way they are hyped sometimes, as fantastic tools for coding and agentic local work. The reality is that they are rather limited, would not do well on a long or complex task, and are prone to fall into loops, forget their tasks, etc. Not mentioned in the article is that they are also rather expensive - not just for the hardware cost, but also electricity. Thes…

My dream would be a local model that can do, say, 80% of the day to day tasks I need; "how does X Handler connect to Y storage?", "commit that feature, but leave out the bits that relate to billing" etc. It would have 99% reliable tool calling - and most importantly - the ability to go "this task is beyond my skills" and refer to a Big Boy Online Model in a gigantic datacenter somewhere. This way all of the simple st…

Claude kind of has this already in their Advisor feature. I don't think I've seen it elsewhere. Open harnesses could add this feature and call out to big boy models when required. It's a really great idea.

Re: Local Qwen isn't a worse Opus, it's a different tool

#83

Post by chartered_stack »

barrkel wrote: post #67

I found it interesting that vLLM was dismissed as slower than llama.cpp. IME vLLM is quite a bit faster than llama.cpp but where it really wipes the floor with it is in batching concurrent load. The downside is that it is dramatically less flexible in terms of tweaking. It gives you very few options for running quantized weights. It takes a lot longer to start up because it optimizes the compute graph. So for single…

One could say: vLLM isn't a worse Llama.cpp, it's a different tool

Re: Local Qwen isn't a worse Opus, it's a different tool

#84

Post by Forgeties79 »

ramon156 wrote: post #63
h05sz487b wrote: post #45

Earlier quoted context omitted.

> It is very much like playing an instrument. Or it is more like playing a slot machine and you imagine the rest.

Instruments are pseudo-random until you know what you're doing. Slot machines are just slot machines

Musical instruments are not random. You’re just doing random inputs. Instruments are consistent, even if the “flavor” and quality varies with different builds.

Playing a B on a saxophone always plays a B.

Re: Local Qwen isn't a worse Opus, it's a different tool

#85

Post by amelius »

dkersten wrote: post #80
glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

> This is not scientific at all, just vibes, YMMV. This is the problem. I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X" , or "model Y should be used in ways Z" . But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive,…

Yes, but benchmarks can be gamed.

Maybe we need better reviewers then?

Re: Local Qwen isn't a worse Opus, it's a different tool

#86

Post by dotancohen »

dkersten wrote: post #80
glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

> This is not scientific at all, just vibes, YMMV. This is the problem. I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X" , or "model Y should be used in ways Z" . But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive,…

Honestly, the differences between AI models always felt to me like the differences between coworkers or job candidates. They don't all share the same strengths and weaknesses - and they all have both good days and bad days.

Realising this made me respect the "I" in "AI" a bit more seriously.

Re: Local Qwen isn't a worse Opus, it's a different tool

#87

Post by eurekin »

> The model is running so hot, that it shoots past the goal and starts looping

later:

> My latest experiment was setting up vLLM (the gold standard for production and concurrent serving) and even with an NVLink (175GBP) and tensor parallelism turned on, it was 3 tokens/second slower than llama.cpp during generation for an equivalent setup.

In all my tests, getting vllm to run is worth it. It was the single biggest thing, that helped for looping issues, agents going whack and losing focus on the task, long context being essentially useless.

FP8 model, unquantized cache in vllm an you have a league better overall experience, with any other stack I tested. Then, you can actually focus on using the model for other things and stop tinkering with settings.

Re: Local Qwen isn't a worse Opus, it's a different tool

#88

Post by mnicky »

weitendorf wrote: post #49
glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

One thing I used to test quite a lot was rerunning the exact same prompt on the same input, or semantically equivalent (in my mind) but differently framed or worded input, and seeing how much they diverged. In particular I’ve done this quite a lot between Sonnet vs Opus and across Qwen models. I recommend everybody do this because you don’t need any special data except what you are already using, and the results will…

If the benefits of using the model you've come to know well outweigh the disadvantages, you can continue using it even after the release of a successor model, right?

Re: Local Qwen isn't a worse Opus, it's a different tool

#89

Post by dotancohen »

weitendorf wrote: post #49
glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

One thing I used to test quite a lot was rerunning the exact same prompt on the same input, or semantically equivalent (in my mind) but differently framed or worded input, and seeing how much they diverged. In particular I’ve done this quite a lot between Sonnet vs Opus and across Qwen models. I recommend everybody do this because you don’t need any special data except what you are already using, and the results will…

  > We’ve been calling some of these “magic words” at work, specific technical terms or references/techniques that you need only mention to get vast improvements in outcome.
Any chance you could share some of these? Seems like something we could all benefit from.

Re: Local Qwen isn't a worse Opus, it's a different tool

#90

Post by dotancohen »

Forgeties79 wrote: post #84
ramon156 wrote: post #63

Earlier quoted context omitted.

Instruments are pseudo-random until you know what you're doing. Slot machines are just slot machines

Musical instruments are not random. You’re just doing random inputs. Instruments are consistent, even if the “flavor” and quality varies with different builds. Playing a B on a saxophone always plays a B.

Saxophone, being a wind instrument was a bad choice. I can definitely tell which student was blowing when hearing a note.

But your analogy remains solid if you substitute e.g. a piano and a reasonably proficient player. A single note would be nearly indistinguishable between players... But a full piece most certainly will sound different.

Post reply on HN