>Local Qwen isn't a worse Opus > looks inside >local Qwen is not "near Opus levels"
Ok I'll bite. What's the contradiction?
Local Qwen isn't a worse Opus, it's a different tool
251–260 of 271 posts
Re: Local Qwen isn't a worse Opus, it's a different tool
#252Earlier quoted context omitted.
No doubt that computers enabled a lot of automation. We can both agree with that. The context was that technology should evolve to fit the humans [not the other way around]. And if contemporary technology didn't have limitations, it would be correct. But it did and humans had to adapt to the computers. Humans had to develop and learn special languages so they could communicate with computers to do all those useful th…
> Humans had to develop and learn special languages so they could communicate with computers to do all those useful things you mentioned. Why? They were limited in understanding (or parsing) human languages You may need to revisit your computation theory courses. Computers are the embodiment of a mathematical model and thus the inputs and outputs are formalized. Do you just hold a pen and words are written automatica…
I don't think the "languages" they said meant specifically "programming languages". In HCI, computer interfaces can be referred as languages as they come with their own affordances and symbolism that is not directly associated with real life: case in point, nowadays, basically no one saves data in diskettes, but we still use them as the "save icon".
Also, I find it funny you mentioned "there's no such thing as words [...] at the computer level". It seems you are the one in the need of a computational theory refresh. Grammars are composed of words, which in turn, are composed of elements of the alphabet set. So, in fact, not only there are words, computers are, above all else, word-processing machines. There are more innacuracies (physical computers being stricly deterministic, needing binary to accomplish inference, etc.), but let's leave it at that, unless you wish to press.
Re: Local Qwen isn't a worse Opus, it's a different tool
#253Earlier quoted context omitted.
Fair enough, that sentence was fairly compressed. I’ve reworded it - the meaning remains the same. The post is not AI generated, I use AI for code generation and write my own articles. Which part of the post are you struggling with? This is a post describing our own experience and journey. Happy to back up any specific claim.
How about your reply here? Was that AI-generated? If not, are you conscious of how much you're starting to sound like AI? Is that something you see as a positive thing, or something you'd like to avoid? I actually find this somewhat interesting, because it seems that a lot of people who weren't comfortable with expressing themselves verbally are feeling more empowered in that area. We're hearing new voices for the fi…
Or maybe it's you who sees agents everywhere... could it be?
Re: Local Qwen isn't a worse Opus, it's a different tool
#254Earlier quoted context omitted.
> Humans had to develop and learn special languages so they could communicate with computers to do all those useful things you mentioned. Why? They were limited in understanding (or parsing) human languages You may need to revisit your computation theory courses. Computers are the embodiment of a mathematical model and thus the inputs and outputs are formalized. Do you just hold a pen and words are written automatica…
I don't think the "languages" they said meant specifically "programming languages". In HCI, computer interfaces can be referred as languages as they come with their own affordances and symbolism that is not directly associated with real life: case in point, nowadays, basically no one saves data in diskettes, but we still use them as the "save icon". Also, I find it funny you mentioned "there's no such thing as words…
> In HCI, computer interfaces can be referred as languages as they come with their own affordances and symbolism that is not directly associated with real life:
There's always jargon and other token words that holds no meaning in other realm of life. Even the alphabet today is mostly arbitrary gliphs.
> Grammars are composed of words, which in turn, are composed of elements of the alphabet set.
Please refer to the formal definition found in wikipedia
https://en.wikipedia.org/wiki/Context-free_grammar#Formal_de...
> There are more innacuracies (physical computers being stricly deterministic, needing binary to accomplish inference, etc.),
I've not said anything about computers being strictly deterministic. And everything is binary at the CPU/GPU level. Even with specialized instructions, you still need to organize them into a proper algorithm and encode it and its data to binary.
Re: Local Qwen isn't a worse Opus, it's a different tool
#255If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…
One thing I used to test quite a lot was rerunning the exact same prompt on the same input, or semantically equivalent (in my mind) but differently framed or worded input, and seeing how much they diverged. In particular I’ve done this quite a lot between Sonnet vs Opus and across Qwen models. I recommend everybody do this because you don’t need any special data except what you are already using, and the results will…
> rerunning [...] but differently framed or worded input, and seeing how much they diverged
I'm surprised how little attention this is getting in today's comments. Open-weights means being able to afford multiple runs, space sampling, critique and synthesis.
Last night, sketching some intro biology content emphasizing cross-cutting concerns, Qwen would get sucked into the Next Generation Science Standards attractor. But nudge it by adding just one similar phrase of Chinese, and most runs ran free (outputting English but for headers with parenthesized Chinese). The multi-lingual LLM "no, not *that* region of latent space".
Re: Local Qwen isn't a worse Opus, it's a different tool
#256Earlier quoted context omitted.
Fair enough, that sentence was fairly compressed. I’ve reworded it - the meaning remains the same. The post is not AI generated, I use AI for code generation and write my own articles. Which part of the post are you struggling with? This is a post describing our own experience and journey. Happy to back up any specific claim.
> Fair enough ... compressed ... ACTION->RESULT ... NEGATION->STATEMENT ... follow up questions. What model are you again?
It is interesting to see the degree to which Claude voice creeps into normal speech. It also leads to some annoying catch-22’s, like rejecting an em-dash where you might have genuinely written one to avoid the appearance of LLM text.
Thankfully I’m lazy and have always used hyphens everywhere so at least I’m safe on that point.
Re: Local Qwen isn't a worse Opus, it's a different tool
#257Earlier quoted context omitted.
Aren't there benchmarks that measure at the harness level as well?
How would you benchmark "agent harness communicates with user clearly" it's 100% a feels measurement.
I mean, in my experience some of this stuff is way closer to table stakes things than that. Like "the tool call didn't get totally confused" more than "did the communication with the user feel good".
Re: Local Qwen isn't a worse Opus, it's a different tool
#258Earlier quoted context omitted.
I don't think the "languages" they said meant specifically "programming languages". In HCI, computer interfaces can be referred as languages as they come with their own affordances and symbolism that is not directly associated with real life: case in point, nowadays, basically no one saves data in diskettes, but we still use them as the "save icon". Also, I find it funny you mentioned "there's no such thing as words…
> In HCI, computer interfaces can be referred as languages as they come with their own affordances and symbolism that is not directly associated with real life: There's always jargon and other token words that holds no meaning in other realm of life. Even the alphabet today is mostly arbitrary gliphs. > Grammars are composed of words, which in turn, are composed of elements of the alphabet set. Please refer to the fo…
> There's always jargon and other token words that holds no meaning in other realm of life. Even the alphabet today is mostly arbitrary gliphs.
Sure, but this is a discussion focused on how humans interact with computers, ergo Human-Computer Interactions, so I'm not sure what's your point. In the end, you don't interact with your computer (in the physical sense) through a 2-key keyboard.
> Please refer to the formal definition found in wikipedia
When I mentioned grammars, I was talking about formal grammars in general. Still, I made a bit of confusion, since formal grammars only define the rules, whereas formal languages are, in one of its definitions, sets over strings/words.
Not that this means much, since the point of grammars is to define languages. As such, grammars (RG/CFG/NG/UG) stipulate the words that a language accepts. Words are important to computers (both in mathematical theory and in material reality).
> I've not said anything about computers being strictly deterministic.
My bad, that was my misreading of "formalized".
> And everything is binary at the CPU/GPU level. Even with specialized instructions, you still need to organize them into a proper algorithm and encode it and its data to binary.
Poor phrasinf on my part, but the "needing binary to accomplish inference" was supposed to be read in isolation. Still, computers do not require binary to operate. There are non-digital computers, both in history and being explored today. There are experiments on using trinary for optimizing LLM inference, for instance.
Re: Local Qwen isn't a worse Opus, it's a different tool
#259If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…
> This is not scientific at all, just vibes, YMMV. This is the problem. I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X" , or "model Y should be used in ways Z" . But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive,…
So, this may not be precisely what you're looking for but it may come close. I've put together a simple site for sharing ratings/opinions on models on a task-specific granularity. https://model.reviews/
The idea is that benchmark score comparisons are useful for a large cross-product comparison across models + their settings, but less useful if you're looking for the best model for . So on this site, each model gets its own page showing the list of tasks that people have rated it on, and the score out of 10 for each task. Common tasks, like coding, will likely be on most/all models, and more niche tasks may only be on a few. It is human moderated (by me only right now).
The corpus is pretty empty right now, so please spread the word if this seems like a useful idea!
Re: Local Qwen isn't a worse Opus, it's a different tool
#260Earlier quoted context omitted.
This is how I feel whenever I see bold all caps instructions in a system prompt or someone claims they conducted "research" and found the magic prompt template that makes the model pay out. Maybe it works some of the time but it isn't a solution that works everytime. It reminds me of people hovering to play a slot machine when someone gets up and it hasn't paid out as if they've solved slot machines. While I don't mi…
Has there been any evidence of a well known provider rerouting to lower quality models? Last I saw, engineers working at OpenAI denied this on HN. I saw that someone set up a tracker that aims to record the performance of the models, and so far it has not shown any statistically significant deviation in performance for Codex, and not yet enough data for Claude: https://marginlab.ai/trackers/codex/
Yes, OpenAI admits they silently reroute sensitive requests to different models for user welfare at least: https://openai.com/index/building-more-helpful-chatgpt-exper...
The implementation was so borked, SamA went back on Reddit and apologised: https://old.reddit.com/r/ChatGPT/comments/1o6jins/updates_fo...
Model re-routing happens for coding tasks too. For example, in OpenAI support pages used to (at least 1 month ago when I checked) mention that if they automatically use a cheaper -mini to accomplish the task behind the scenes, you’ll be charged -mini prices even if you selected a more expensive model. I just checked again and they’ve removed it, but there’s probably archives.
Finally, even if they’re the same weights, you don’t know what quantisation you’re running at. Adaptive quantisation based on load (given workday peaks), or similar techniques, have been happening since the ChatGPT 3.5 days; the techniques are probably more advanced now.