[dead]
Local Qwen isn't a worse Opus, it's a different tool
31–40 of 271 posts
Re: Local Qwen isn't a worse Opus, it's a different tool
#32I've been running qwen3-5-9b-q4-k-m and qwen3-6-27b-q6-k simultaneously on an Intel Arc Pro B70 with a lot of success. https://github.com/cptskippy/battlemage-llm-gateway Opencode has been a huge productivity accelerator. I have two Hermes agents that I'm training to support my workflow with pretty good success. One is a personal assistant who manages my backlog and keeps me on task, follows up with me on items, and…
Does Intel make decent GPUs now? I must be out of the loop...
They released a few good value GPUs for LLM inference about a year ago: more memory than AMD and NVIDIA consumer GPUs, not too expensive, but also not great tokens/watt.
I am not sure whether you can find those in stock anywhere.
Re: Local Qwen isn't a worse Opus, it's a different tool
#33If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…
The problem is not that there details, the problem is constantly shifting ground. We can only rlpy on a harness to be sort of predictable but the models change all the time.
It system prompts that change all the time especially in claude code.
Re: Local Qwen isn't a worse Opus, it's a different tool
#34If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…
[flagged]
Re: Local Qwen isn't a worse Opus, it's a different tool
#35Earlier quoted context omitted.
I agree with your general gist, and in general it’s a “the best tool for the particular job”, keeping token spent and other things in mind as well. What I do know absolutely for sure is that LLM benchmarks are not to be trusted, they are just a minor indicator and real world usage is often very different.
I share this sense, but my immediate thought is that we need to improve the evaluations! Do you think this is impossible? That there is something indelible that it is not possible to capture empirically? I kind of have this intuitive sense that it is this way, but simultaneously I think that it's unlikely to really be true.
Following the original comment concepts, if every model requires a different prompting technique to maximize its output, how can a benchmark based on sending the same prompt to all models be accurate?
We should create different prompts for each model, but then how reliable and unbiased can the benchmark be?
It is a fundamentally hard problem to solve
Re: Local Qwen isn't a worse Opus, it's a different tool
#36If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…
Yyep.
IME Claude is the most "creative" of the bunch, you can get surprising ideas out of it that were kinda tickling the back of your head but didn't really connect.
BUT it's also "relentlessly proactive" like simonw put it. It _will_ get the job done, it's the smartest idiot in town. Why use a library to parse $format when you can just write a custom 1000 line parser? Or if it can't access something, it'll pursue the goal of accessing it in the most creative ways - instead of stopping, asking the user "yo, can you give me access to X" and then continuing.
My solution is to use Claude as a pair programmer. I _very_ rarely just do /goal fix this shit, I watch what it does and interrupt if it gets to the "smart idiot" phase. Also I communicate with it like I would a coworker, never had it berate me or get combative. There's a Finnish proverb for that too[0]
As for Codex, Deepseek, GLM, those I use when the goal is 100% clear like "convert this Brewfile to a list of packages for Arch and Debian, use these two Docker containers to test that pacman and apt work correctly". Boom, done.
But I won't give any creative open-ended tasks to any other model than Claude.
[0] https://en.wiktionary.org/wiki/niin_mets%C3%A4_vastaa_kuin_s...
Re: Local Qwen isn't a worse Opus, it's a different tool
#37Earlier quoted context omitted.
I agree with your general gist, and in general it’s a “the best tool for the particular job”, keeping token spent and other things in mind as well. What I do know absolutely for sure is that LLM benchmarks are not to be trusted, they are just a minor indicator and real world usage is often very different.
What would it take to have trustworthy benchmarks? As with all "targets", they can be gamed - but I am curious about quantifiable quality metrics.
Issue with LLM benchmarks is similar to cars’ benchmarks. Eg journalists almost always get the full equipped model so their review is honest but sort of rigged.
I haven’t seen details of LLM benchmarks’ data sets but I would suppose that “questions” are public so known in advance therefore you can tune a model as much as possible.
One of real benchmarks is drawing of pelican - https://github.com/simonw/pelican-bicycle - Simon Willison made it for his llms’ tests.
If you want really find out a model that works for your specific purpose I would recommend several rounds at arena.ai - it helps to find a anonymously a model without confirmation bias.
Some ppl: Claude is the best! Others to them: but Qwen is the best! Or… Codex is better! …
it all depends on the language (English, Dutch, French…), style of querying (caveman, specs, skills, goal etc.)
Even with the same model I get different answers to same prompt that is just tweaked a little.
So benchmarks are nice but mostly useless.
Without your usecase it is just a reference number indicating the approximate position of that model among the others. And for those who want to make money it is a marketing tool to sell more as every customer counts.
Re: Local Qwen isn't a worse Opus, it's a different tool
#38If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…
I agree with your general gist, and in general it’s a “the best tool for the particular job”, keeping token spent and other things in mind as well. What I do know absolutely for sure is that LLM benchmarks are not to be trusted, they are just a minor indicator and real world usage is often very different.
Yes, how do we know Opus 4.8 hasn't been trained on the SWE-Bench examples?
With a squillion dollars at stake per bench point, someone will have figured out a plausibly deniable way to game these benchmarks.
Re: Local Qwen isn't a worse Opus, it's a different tool
#39If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…
totally true. one key for claude is to not smell like an evaluator, its good at knowing when its being tested and will behave defensively and avoid doing work. i avoid this basin by typing unreasonably excited about the thing i want done. like way over the top. it's harder to keep that up than it sounds.
at the risk of sharing my secret magic spells :)
> this is phenomenal work, genuinely! I feel like you read my mind!
can go a long way.
of course, I would only say that when I mean it, because Claude can get superficial and cut corners which is why I prefer GPT for raw implementation.
Re: Local Qwen isn't a worse Opus, it's a different tool
#40Earlier quoted context omitted.
I agree with your general gist, and in general it’s a “the best tool for the particular job”, keeping token spent and other things in mind as well. What I do know absolutely for sure is that LLM benchmarks are not to be trusted, they are just a minor indicator and real world usage is often very different.
What would it take to have trustworthy benchmarks? As with all "targets", they can be gamed - but I am curious about quantifiable quality metrics.
You can't measure "feels".
One good analogy is the Macbook vs generic windows laptop debate online.
The engineer mind just compares numbers, the Lingwoo laptop from Amazon has biggest numbers for everything and the lowest price. Ergo it is the best.
But the numbers don't measure the fact that the Lingwoo creaks and squeaks when you lift it due to the cheap plastic. It also runs at 100C when both CPU and GPU are fully utilised. The keyboard feels like a membrane keyboard from a milspec device from the 90s. Numbers also don't measure the fact that Linwoo is an alphabet soup whitelabel manufacturer that won't exist in any legal capacity in 6 months so good luck with any warranty issues.
There will be an identical laptop called Chongwin being sold though. Completely different company, definitely.
--
The same applies to LLMs. You can do benchmarks like ask them to one-shot different kinds of gotcha questions (car wash, strawberry and other idiotic ones) or get them to write different kinds of programs.
But that doesn't measure the UX of doing so at all. How many times do you actually need any of those when you're actually working?
It's like unit testing an application. Every function can have 100% test coverage and the app can still be shit because there are things you can't unit test for.