Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local Qwen isn't a worse Opus, it's a different tool

blog.alexellis.io

Post reply on HN
Print view View original

131–140 of 271 posts

Re: Local Qwen isn't a worse Opus, it's a different tool

#131

Post by stego-tech »

I still believe that the strength of AI is when it can be applied locally in a secure and private manner, rather than yet another cloud-based service you must pay for indefinitely even as it gets progressively worse to satiate the greed of corporate shareholders.

ChatGPT and Anthropic will never, ever get me to tie my Health Data to their systems, but I still believe in the capabilities of AI in identifying patterns from data I would otherwise overlook, and sorely want a local-only ecosystem where I can expose this data safely, privately, and securely to something like Qwen or Gemma for processing.

Same goes for Smart Homes, and Personal Assistants. The corporate approach of letting Company A access your data stored at Company B and processed by Companies D and E while also sold to Advertisers and Data Brokers with no way for you to extract or view it on your local hardware - just isn’t tenable for these sorts of intimate use cases. I want my data to be owned and controlled and exposed on my terms, to be used to improve my life first rather than someone else’s bottom line. I want technology to give me back more of my time and improve my outcomes again, and I’ve been burned enough by Big Tech in the past that I flatly reject any presumption of nobility or public good from their AI-as-a-Service business model.

The capability is there, and I definitely think the folks working to build local tooling that supports and unlocks the potential for local models are the ones in the right. I love seeing what they build.

Re: Local Qwen isn't a worse Opus, it's a different tool

#132

Post by Iolaum »

eurekin wrote: post #87

> The model is running so hot, that it shoots past the goal and starts looping later: > My latest experiment was setting up vLLM (the gold standard for production and concurrent serving) and even with an NVLink (175GBP) and tensor parallelism turned on, it was 3 tokens/second slower than llama.cpp during generation for an equivalent setup. In all my tests, getting vllm to run is worth it. It was the single biggest th…

Why unquantized instead of Q8 ?

Re: Local Qwen isn't a worse Opus, it's a different tool

#133

Post by malwrar »

cassianoleal wrote: post #125
coldtea wrote: post #103

Earlier quoted context omitted.

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

They are not human. Humans have names, faces, voices, personality, a personal history, family, care for whatever they call their community. With humans it's actually good and worthwhile to create and strengthen connections. With an LLM, that's psychosis.

These things passing the Turing Test makes anthropomorphizing their behavior awkward, but don’t forget it’s just an analogy to communicate an experience. If you convey a certain written voice to these models in your input, you get a somewhat consistent end effect. I think that’s all that is being communicated.

Re: Local Qwen isn't a worse Opus, it's a different tool

#134

Post by coldtea »

scotty79 wrote: post #126
cassianoleal wrote: post #125

Earlier quoted context omitted.

They are not human. Humans have names, faces, voices, personality, a personal history, family, care for whatever they call their community. With humans it's actually good and worthwhile to create and strengthen connections. With an LLM, that's psychosis.

If you have a toolbox full of similar but different tool getting to know them is a prudent thing to do, not a psychosis. There's no connection because the tool is immutable (except for adjustments you made) but you do develop a specific relation with that tool. Some people even love some of their tools at some level. And if humans are anything, they are tool users.

>If you have a toolbox full of similar but different tool getting to know them is a prudent thing to do, not a psychosis

Can be both. Use of some tools like LLMs might be more inducing psychosis than others like plain compilers or hammers.

>And if humans are anything, they are tool users.

To the point of self-destruction sometimes.

Re: Local Qwen isn't a worse Opus, it's a different tool

#135

Post by yeer2 »

coldtea wrote: post #103
dkersten wrote: post #80

Earlier quoted context omitted.

> This is not scientific at all, just vibes, YMMV. This is the problem. I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X" , or "model Y should be used in ways Z" . But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive,…

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

This is so dumb and goes against all the principles that enabled computers and smartphones to achieve wide adoption - the technology should evolve to fit the human. Not the other way around.

Re: Local Qwen isn't a worse Opus, it's a different tool

#136

Post by sleepyeldrazi »

sanderjd wrote: post #15
zmmmmm wrote: post #9

That's a great write up. The one thing I feel it seems to under estimate is the likelihood of improvement. Even the authors acknowledge it's not even worth comparing local models from a year ago to what we have now. In fact, people widely see Opus 4.5 in November last year - 8 months ago - as the first time agentic coding became viable broadly viable even with frontier hosted models. So why would we lock in hard on a…

Right. Opus 4.5 8 months ago, good enough for agentic coding. How far behind that are open weight models? More than 8 months? But how much more? When will they reach Opus 4.5 level? A few months from now? A year from now? Never?

Opus also has a deeply ingrained personality that always de-rails sneakily into what it's taught, not what the user intends. This is good if the user doesn't know the details of the work they need performed and a huge time waste when the user knows exactly how something needs to be implemented.

I have found claude models, especially fable, to be impossible to work with when the work requires reading papers from days ago and reasoning on top of the findings in it. I have multiple long sessions with opus (not as many with fable as it got taken down quickly) where it keeps fighting me on problems, sayings "that's not how it works" / "that is not possible", followed by me linking the paper (after i've told it to actually read up on the latest research in this field), and it hits me with the usual "You were right.". If your workflow is using the exact tools, frameworks, git layouts that claude expects, it can be magical, yes. But it is very heavily optimized to never say 'I am not sure' (as that gives 'bad vibes') and instead lean on its (nowadays with the speed of things DOE) knowledge to formulate a reasonable sounding answer, dissectible only if you already know the answer beforehand (which defeats the purpose of using it in the first place).

Qwen3.6 27B (the only Knowing what is in context, what should and shouldn't be there, and how to manage it for the specific model you are using (as every model, even in the same family, behaves differently to differently worded prompts) is what makes or breaks them. They are just auto-complete, they complete text based on what is already there, it's not magic.

So yes, while this small open-weights models are not opus 4.5, it's good precisely because if that, because it is a good tool and a bad 'coworker replacement'. If you want the latter, kimi is already there, it has started to not believe the user and do what it was taught just like claude models (which is helpful when you don't care about implementation specifics or performance/security). GLM models (mostly 5.1, i haven't tested 5.2 extensively yet) have fixed a lot of low-level programming issues I've had that opus just walks in circles and writes reports that "it doesn't/can't work". That is to say, open-weights, in many cases, have already surpassed Opus. I can't comment on gpt 5.5, but while I used 5.4, it also performed a lot more tasks without being fussy than opus 4.6/4.7.

Re: Local Qwen isn't a worse Opus, it's a different tool

#137

Post by coldtea »

gib444 wrote: post #121
coldtea wrote: post #103

Earlier quoted context omitted.

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

No, I won't anthropomorphise LLMs.

If there was anything that made sense to anthropomorphise it would be a machine meant to mimic talking, thinking and answering like a human, one that even passes the Turing test.

When we built the idea that anthropomorphising is wrong, we meant when doing it for rocks or trees or thunders or deer or some such.

Re: Local Qwen isn't a worse Opus, it's a different tool

#138

Post by victorbjorklund »

hodgehog11 wrote: post #74
h05sz487b wrote: post #45

Earlier quoted context omitted.

> It is very much like playing an instrument. Or it is more like playing a slot machine and you imagine the rest.

A poor analogy depending on the setting because you can't adjust the odds with a slot machine, and the ROI is negative by design. If that's your experience, yeah, I wouldn't use an LLM either.

Pretty sure most modern slot machines are digital and you could adjust the odds (even to a positive EV) if you change the code.

Re: Local Qwen isn't a worse Opus, it's a different tool

#139

Post by Wowfunhappy »

yeer2 wrote: post #135
coldtea wrote: post #103

Earlier quoted context omitted.

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

This is so dumb and goes against all the principles that enabled computers and smartphones to achieve wide adoption - the technology should evolve to fit the human. Not the other way around.

I mean, like, you can lament the state of the world all you want. It is what it is. Of course the AI labs would also like to make their models more consistent, but it's not how the technology works. They're black boxes to everybody.

Re: Local Qwen isn't a worse Opus, it's a different tool

#140

Post by eurekin »

Iolaum wrote: post #132
eurekin wrote: post #87

> The model is running so hot, that it shoots past the goal and starts looping later: > My latest experiment was setting up vLLM (the gold standard for production and concurrent serving) and even with an NVLink (175GBP) and tensor parallelism turned on, it was 3 tokens/second slower than llama.cpp during generation for an equivalent setup. In all my tests, getting vllm to run is worth it. It was the single biggest th…

Why unquantized instead of Q8 ?

Noticed few cliffs. Sometimes it was a spurious stop (had to write "go on" or "continue" to restart), othertimes it was randomly saying: "Oh the user wants [the thing we already resolved]" and goes back in history. Cleared all out on fp16
Post reply on HN