Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local Qwen isn't a worse Opus, it's a different tool

blog.alexellis.io

Post reply on HN
Print view View original

121–130 of 271 posts

Re: Local Qwen isn't a worse Opus, it's a different tool

#121

Post by gib444 »

coldtea wrote: post #103
dkersten wrote: post #80

Earlier quoted context omitted.

> This is not scientific at all, just vibes, YMMV. This is the problem. I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X" , or "model Y should be used in ways Z" . But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive,…

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

No, I won't anthropomorphise LLMs.

Re: Local Qwen isn't a worse Opus, it's a different tool

#122

Post by Forgeties79 »

sanderjd wrote: post #14
stingraycharles wrote: post #10

Earlier quoted context omitted.

I agree with your general gist, and in general it’s a “the best tool for the particular job”, keeping token spent and other things in mind as well. What I do know absolutely for sure is that LLM benchmarks are not to be trusted, they are just a minor indicator and real world usage is often very different.

I share this sense, but my immediate thought is that we need to improve the evaluations! Do you think this is impossible? That there is something indelible that it is not possible to capture empirically? I kind of have this intuitive sense that it is this way, but simultaneously I think that it's unlikely to really be true.

The reason we can’t capture it empirically is that nobody truly knows exactly what we are supposed to be using these tools for or how they are going to operate. We are still fitting squares into holes with them. We are told to treat them like some bespoke tool for coding, shopping, tech-support, etc. But it is not actually purpose built for any of these things.

When I use a calculator, I know exactly what it does and what it is supposed to do. It always gives me a verifiable, predictable result. If I input “8+8” 10,000x it will give me “16” 10,000x outside of incredibly fringe edge cases/bugs. I can’t say the same for LLMs

Re: Local Qwen isn't a worse Opus, it's a different tool

#123

Post by dreambuffer »

coldtea wrote: post #103
dkersten wrote: post #80

Earlier quoted context omitted.

> This is not scientific at all, just vibes, YMMV. This is the problem. I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X" , or "model Y should be used in ways Z" . But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive,…

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

Please do not think of LLMs like human helpers, that is a recipe for long term sociopathy.

Re: Local Qwen isn't a worse Opus, it's a different tool

#124

Post by evntdrvn »

weitendorf wrote: post #49
glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

One thing I used to test quite a lot was rerunning the exact same prompt on the same input, or semantically equivalent (in my mind) but differently framed or worded input, and seeing how much they diverged. In particular I’ve done this quite a lot between Sonnet vs Opus and across Qwen models. I recommend everybody do this because you don’t need any special data except what you are already using, and the results will…

One thing that I learned when doing raw API LLM usage is how drastically the results can vary call per call with exactly the same input. I think that on average, people using agents underestimate the variation in results from a given turn command are, and so overindex on "X technique worked well" or "if I do Y then this will happen" or even "it did Z task well last time so it will this time too" or "{Model} is great at {thing}"

Re: Local Qwen isn't a worse Opus, it's a different tool

#125

Post by cassianoleal »

coldtea wrote: post #103
dkersten wrote: post #80

Earlier quoted context omitted.

> This is not scientific at all, just vibes, YMMV. This is the problem. I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X" , or "model Y should be used in ways Z" . But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive,…

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

They are not human. Humans have names, faces, voices, personality, a personal history, family, care for whatever they call their community.

With humans it's actually good and worthwhile to create and strengthen connections. With an LLM, that's psychosis.

Re: Local Qwen isn't a worse Opus, it's a different tool

#126

Post by scotty79 »

cassianoleal wrote: post #125
coldtea wrote: post #103

Earlier quoted context omitted.

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

They are not human. Humans have names, faces, voices, personality, a personal history, family, care for whatever they call their community. With humans it's actually good and worthwhile to create and strengthen connections. With an LLM, that's psychosis.

If you have a toolbox full of similar but different tool getting to know them is a prudent thing to do, not a psychosis. There's no connection because the tool is immutable (except for adjustments you made) but you do develop a specific relation with that tool. Some people even love some of their tools at some level.

And if humans are anything, they are tool users.

Re: Local Qwen isn't a worse Opus, it's a different tool

#127

Post by tekne »

cassianoleal wrote: post #125
coldtea wrote: post #103

Earlier quoted context omitted.

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

They are not human. Humans have names, faces, voices, personality, a personal history, family, care for whatever they call their community. With humans it's actually good and worthwhile to create and strengthen connections. With an LLM, that's psychosis.

To be fair: a voice, personality, and personal history sounds a lot like training data.

I don't think LLMs are people in any sense, at least as they're constructed now -- but they very much have what we would call "culture" and "personality" in suitably alien forms.

This is not the same as, e.g., feelings, experience, or humanity, or actual opinions or ideas (versus essentially "distilled vibes") and I feel that AI will more and more force us to confront that (including if new AIs are ever developed that may have the latter, as well!)

Re: Local Qwen isn't a worse Opus, it's a different tool

#128

Post by tekne »

palata wrote: post #96
dotancohen wrote: post #90

Earlier quoted context omitted.

Saxophone, being a wind instrument was a bad choice. I can definitely tell which student was blowing when hearing a note. But your analogy remains solid if you substitute e.g. a piano and a reasonably proficient player. A single note would be nearly indistinguishable between players... But a full piece most certainly will sound different.

While I agree with you, I think it's diverging from the initial point. The original take was "LLMs are very much like playing an instrument". I think they are very much NOT like playing an instrument. While different musicians will produce different results, one musician won't get drastically different results on different days or when trying a different "copy" of the same instrument. If you can play the violin on yo…

Will you?

That's only if both violins are tuned the same way, and one must continually tune them lest they get out of sync.

Similarly, an LLM can be extremely consistent if tuned properly -- indeed, if you fix the weights and settings, they can be made "essentially deterministic" for many prompts!

Re: Local Qwen isn't a worse Opus, it's a different tool

#129

Post by eurekin »

trey-jones wrote: post #109
eurekin wrote: post #87

> The model is running so hot, that it shoots past the goal and starts looping later: > My latest experiment was setting up vLLM (the gold standard for production and concurrent serving) and even with an NVLink (175GBP) and tensor parallelism turned on, it was 3 tokens/second slower than llama.cpp during generation for an equivalent setup. In all my tests, getting vllm to run is worth it. It was the single biggest th…

I'm really curious about this, not because I disagree, but because I want to avoid agents going whack. Are you running vllm for yourself only, or a for a team, or for an application, etc? And do you feel there is a minimum hardware requirement for vllm to be useful in this way? My weekend project is going to be building a home inference server (from ancient datacenter parts) and I'm still massaging in my head what th…

If I started today, with building a server, I'd jump right into verified set-ups and writeups, like this one:

https://github.com/noonghunna/club-3090

You can find info about running a patched version of vllm for 1x24gb, 2x and 4x. There's also quite a few "blackwell" subreddits, where people seem to share a lot of substantial information, if you're going the 6000 route.

Re: Local Qwen isn't a worse Opus, it's a different tool

#130

Post by j-bos »

scotty79 wrote: post #126
cassianoleal wrote: post #125

Earlier quoted context omitted.

They are not human. Humans have names, faces, voices, personality, a personal history, family, care for whatever they call their community. With humans it's actually good and worthwhile to create and strengthen connections. With an LLM, that's psychosis.

If you have a toolbox full of similar but different tool getting to know them is a prudent thing to do, not a psychosis. There's no connection because the tool is immutable (except for adjustments you made) but you do develop a specific relation with that tool. Some people even love some of their tools at some level. And if humans are anything, they are tool users.

Yeah, AI tools bring software developers closer to the messy real world where 0 and 1 aren't always exactly 0 and 1.
Post reply on HN