Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local Qwen isn't a worse Opus, it's a different tool

blog.alexellis.io

Post reply on HN
Print view View original

171–180 of 271 posts

Re: Local Qwen isn't a worse Opus, it's a different tool

#171

Post by willtemperley »

stingraycharles wrote: post #163
willtemperley wrote: post #38

Earlier quoted context omitted.

Yes, how do we know Opus 4.8 hasn't been trained on the SWE-Bench examples? With a squillion dollars at stake per bench point, someone will have figured out a plausibly deniable way to game these benchmarks.

Ehr, the SWE bench examples are particularly horrible as those are just publicly available historical PRs. So if the models are trained on GitHub data, it will be included. So almost by design that particular benchmark is tainted, and benchmarks recall rather than reasoning.

Wow that's worse than I thought, and breaks the number one rule of machine learning: you don't train the model with your test dataset.

Re: Local Qwen isn't a worse Opus, it's a different tool

#172

Post by duckmysick »

yeer2 wrote: post #135
coldtea wrote: post #103

Earlier quoted context omitted.

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

This is so dumb and goes against all the principles that enabled computers and smartphones to achieve wide adoption - the technology should evolve to fit the human. Not the other way around.

I'd argue the opposite. Technology in the past few decades was (is) limited and humans had to adapt to it.

We communicate with other humans using voice and three dimensional hand gestures. To use computers and early phones we had to learn to operate new input devices: keyboards and mice. Later with touchscreens we moved to two dimensional hand (finger) gestures. We're barely making voice commands work with our devices just recently.

Then, a large number of humans are figuratively tethered to their desks because the devices need power and stable internet connection. Mobile devices break this relationship a bit but you still need to charge them and be close to some sort of access point. In any case, the devices encourage sitting in one place for hours at time.

And this is just computers and smartphones. Humans adapted their entire lifestyles and transformed the landscape to cater to cars.

Re: Local Qwen isn't a worse Opus, it's a different tool

#173

Post by mahidhar »

coldtea wrote: post #103
dkersten wrote: post #80

Earlier quoted context omitted.

> This is not scientific at all, just vibes, YMMV. This is the problem. I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X" , or "model Y should be used in ways Z" . But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive,…

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

Well, unlike a human, I cannot expect any these LLMs to take any ownership of the work they do. I cannot expect any given model and version (sonnet 4.6) to learn, improve and adapt over time. I cannot expect it's limitations to ever go away at the model level. So it is not like a human in most ways that I actually care about.

That said, I can't wait for LLMs to stop being AI and start being just another tool. Anything cursed with the "AI" label seems to go through this mess. In the earlier AI cycles, rules engines were considered "human-ish" and got hyped up, but today we just see then as just another tool available to us, and we're better off for it.

Re: Local Qwen isn't a worse Opus, it's a different tool

#174

Post by CuriouslyC »

andai wrote: post #153
glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

I asked GLM 5.2 for a HTML5 port of my old C#/XNA game. It ported all the code exactly (except for operator overloading, which doesn't exist in JS), and added more code to make the code work. I asked Claude Sonnet 4.6 for the same thing, and Claude's version was like if the game had been written in JS originally. Also, for some reason it made it a single HTML file, removed all assets, dynamically generated graphics a…

What you've described is Claude's "secret sauce" and the reason some people love it and some people hate it. It's not really possible to turn off, you can try to prompt against it but it's not reliable, the solution is to use Claude when you want that behavior and other models when you don't.

Re: Local Qwen isn't a worse Opus, it's a different tool

#175

Post by yunohn »

dkersten wrote: post #80
glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

> This is not scientific at all, just vibes, YMMV. This is the problem. I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X" , or "model Y should be used in ways Z" . But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive,…

> a product sheet showing what each models strengths an weaknesses are

This presumes that the labs themselves know how well their models perform. But all they have are overtuned benchmarks and hype vibes.

Re: Local Qwen isn't a worse Opus, it's a different tool

#176

Post by layer8 »

tekne wrote: post #128
palata wrote: post #96

Earlier quoted context omitted.

While I agree with you, I think it's diverging from the initial point. The original take was "LLMs are very much like playing an instrument". I think they are very much NOT like playing an instrument. While different musicians will produce different results, one musician won't get drastically different results on different days or when trying a different "copy" of the same instrument. If you can play the violin on yo…

Will you? That's only if both violins are tuned the same way, and one must continually tune them lest they get out of sync. Similarly, an LLM can be extremely consistent if tuned properly -- indeed, if you fix the weights and settings, they can be made "essentially deterministic" for many prompts!

The difference is that a violin player can predict how the known violin will behave under all relevant circumstances, will know how to get the right tone out of it, while you’re generally unable to predict the adequacy of output of even a deterministic LLM. You can’t practically reason about how varying the input to the LLM will ensure the adequacy of its output, while the violin player is perfectly able to do so for the violin.

This is because LLMs have aspects of chaotic dynamical systems, where small changes in initial conditions can lead to vastly different outcomes. That property is independent from nondeterminism.

Re: Local Qwen isn't a worse Opus, it's a different tool

#177

Post by saint-evan »

mnicky wrote: post #88
weitendorf wrote: post #49

Earlier quoted context omitted.

One thing I used to test quite a lot was rerunning the exact same prompt on the same input, or semantically equivalent (in my mind) but differently framed or worded input, and seeing how much they diverged. In particular I’ve done this quite a lot between Sonnet vs Opus and across Qwen models. I recommend everybody do this because you don’t need any special data except what you are already using, and the results will…

If the benefits of using the model you've come to know well outweigh the disadvantages, you can continue using it even after the release of a successor model, right?

Yes! That's exactly true. I have a very real experience on this. I got introduced to Anthropic's family of models with Claude3.5. I fell in love with the specific personality of Sonnet, the model. I can't remember if back then Opus wasn't public yet but I remember very clearly trying out Opus several times when it became touted as best-in-class and actually recoiling from the foreign feel of the Opus model. I remember very well that my problem was that it was way too eager and pretty hard to steer. I returned to Sonnet and I've used ONLY Sonnet ever since. I have/had access to Fable and Opus4.8 but I never once tried them. In the early days with Sonnet3/4.5, I bought ChatGPT, I also remember thinking that it was a great teacher but a lazy coder. You'd get the scaffolding and then '# rest of code block' not full implementation so unless you wanted to learn the concept, weigh trade-offs, ask clarifying questions or jump into a rabbit hole... You had to go code it yourself. ChatGPT generally as a model is a very good teacher so much so that the free version is enough and I use the free in combination with the most advanced Sonnet model for actual SWE day to day. And whenever there's an Opus release I'm actually very excited because it means there's a smarter Sonnet model OTW. I'll actually be veryyy very sad if the Sonnet line gets sunset. There has been no Sonnet upgrades since even as other family lines get improved.

Do note that I only use LLMs in the ChatUI, I never use agents. I don't believe having a blackbox codebase managed by entities with a half-life of 'delete conversation' or 200k tokens is a responsible idea. In ChatUI, I lay the ground rules, kill assumptions about our working relationship, give it foundational context on the problem and codebase we're working on, explain the problem and then we have a conversation about it and I gradually disclose more logically context as it becomes relevant. So, to directly answer your question, maybe I'm missing out on a ton of upside by not using the absolute best but I'd say familiarizing yourself with a specific model has all the benefits of having a human friend you've grown up with... except your buddy's a savant and would absolutely love to help!

Re: Local Qwen isn't a worse Opus, it's a different tool

#178

Post by CuriouslyC »

bandrami wrote: post #120
glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

I think this goes beyond "vibes" to cargo-culting. It's why nobody's ever able to actually show ROI from LLMs

It's hard to actually show ROI from any programming methodology or tool. You can show ROI from a product or feature, but the tool/methodology is a multiplier on the velocity of creating that which is not directly observable.

Re: Local Qwen isn't a worse Opus, it's a different tool

#179

Post by skipants »

I feel like it's the Emperor's new clothes reading this article and seeing the praise it's getting. This sentence doesn't even make sense:

> These products use very low level Linux primitives like containers, Kubernetes, Firecracker microVMs, and networked protocols.

Out of anything that is a "low level linux primitive" I could maybe argue that networking? protocols fit the bill.

And it's obviously fully AI-generated! Which I wouldn't even care about if I could actually trust the content, which I can't!

Re: Local Qwen isn't a worse Opus, it's a different tool

#180

Post by sanderjd »

theshrike79 wrote: post #41
sanderjd wrote: post #14

Earlier quoted context omitted.

I share this sense, but my immediate thought is that we need to improve the evaluations! Do you think this is impossible? That there is something indelible that it is not possible to capture empirically? I kind of have this intuitive sense that it is this way, but simultaneously I think that it's unlikely to really be true.

We shouldn't just measure the power of the raw LLM, harnesses matter more and more. It's like taking the engine out a each car, putting it to a test bed and running it and then making a decision whether the car is good or bad based on the graphs the test bed provided. You might have the best engine in the world, but if you put it in a shit car, the result is still bad. The seats are squeaky plastic, the infotainment…

Aren't there benchmarks that measure at the harness level as well?
Post reply on HN