hope open source continues to improve
GLM 5.2 beats Claude in our benchmarks
341–350 of 559 posts
Re: GLM 5.2 beats Claude in our benchmarks
#342Earlier quoted context omitted.
GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…
If a good SWE is $150/hour, does the model cost actually matter? Surely you'd be willing to spend $10/hour to make that SWE 20% more productive? The model cost is still much less than the salary.
With Claude Code Ultrathink, I used 3 million tokens in 20 minutes. At API prices, that would be around 30$. So 90$/h. Model cost is not that much lower.
Re: GLM 5.2 beats Claude in our benchmarks
#343Earlier quoted context omitted.
> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…
> most halfway decent models can write damn good code for a fraction of the price. The difference is how the model is used. With Opus you can give it a long-horizon task (eg build an entire feature) and it will plan it out and implement it and almost always stay on task. This is what people mean when they say "agentic tasks" With the lessor models the code is fine, but they need something else to plan what needs to b…
My whole point is that I don't want it to build an entire feature from one prompt. At most, I want to work with an agent to nail down the spec and then work with an agent that orchestrates the implementation via other agents, same for testing, etc. None of that requires frontier capabilities, it requires a little bit of work on a harness, a little bit more of my input, a little more of my brainpower. I _want_ to build tools that make it work better and don't change when the CC team gins up some default for their harness and foists it on me. I don't see that as a tradeoff at all and I think engaging in my work process more than fire and forget (and literally always in my experience fix stuff later) is more fun and rewarding once the 'holy shit this is now possible' high wears off. Doubly so once the frontier model gets nerfed mid-cycle and now I have to undo the mess because they released v*.x++ and I fell for it again by trusting it to do these agentic tasks without my involvement.
Re: GLM 5.2 beats Claude in our benchmarks
#344open-weight models routinely match or even outperform previous-generation proprietary APIs
Re: GLM 5.2 beats Claude in our benchmarks
#345Earlier quoted context omitted.
> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…
I've landed in a similar place by reducing effort and cutting up tasks. I find that more exacting specifications to the models, yield significantly less need for "effort". Combining each with multjple git worktrees and an integration branch for the current worktrees themselves has yielded incresible results. This also allows me to play with, and mix codex, claude cli, and others. This is my happy spot for the last tw…
Yeah this is sounds close to my workflow and its good to hear you've find a nice flow too! It frees me up to spend that effort on doing more things in parallel and focusing way more on the specs which is usually a good idea anyway.
Re: GLM 5.2 beats Claude in our benchmarks
#346Earlier quoted context omitted.
So... how's that any different from using American stuff for those of us in the rest of the world? Over the last decade, the US has been way more unreliable than China. There's been a near constant negative impact from the US doing something. At least with China, we are very good at winning trade wars with them here in Australia.
You might feel differently if you were a Filipino or Vietnamese fisherman whose family relied on the income from the stocks of the South China Sea, or a Uighur person living in Western China, or a Ukrainian soldier who has to deal with drones built with Chinese components, or a democracy advocate in Hong Kong, or arguably, a person who had plans for 2020-2021. Or, on a more local note, an Australian automotive worker…
> Or, on a more local note, an Australian automotive worker who worked for a company that figured out 10 years ago that they wouldn't be able to pay him a decent wage, compete with the then-upcoming Chinese EVs, and remain profitable.
I don't understand what your point is? This seems like a perfect example of comparative advantage - Australia can produce iron ore cheaper than anywhere else in the world and even when China launched a trade war against Australia the Australian economy kept growing.
There wasn't even any bump in unemployment from the closing of the car industry.
Once that trade war was settled, Australia got cheaper cars, China got cheaper iron ore and both economies won.
The rational behavior on both parts there is in stark contrast to current US policy, which is unpredictable and capricious.
> You might feel differently if you were a Filipino or Vietnamese fisherman whose family relied on the income from the stocks of the South China Sea, or a Uighur person living in Western China, or a Ukrainian soldier who has to deal with drones built with Chinese components, or a democracy advocate in Hong Kong, or arguably, a person who had plans for 2020-2021.
This seems like a random list of complaints about China and I agree with them in general.
I think you'll find most major powers have similar complaints. There certainly are against the US - I think you might find that both the Philippines and Vietnam(!) have fairly mixed feelings about the US for example.
Re: GLM 5.2 beats Claude in our benchmarks
#347Earlier quoted context omitted.
GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…
If a good SWE is $150/hour, does the model cost actually matter? Surely you'd be willing to spend $10/hour to make that SWE 20% more productive? The model cost is still much less than the salary.
I don’t think any engineers who cost $150/hr are having their productivity moved by 20% depending on a $10/hr gap between models on or near the frontier.
Most of the gains right now come from tooling and process and any big post 2025 language model. The specific model isn’t that important right now.
Re: GLM 5.2 beats Claude in our benchmarks
#348Earlier quoted context omitted.
So... how's that any different from using American stuff for those of us in the rest of the world? Over the last decade, the US has been way more unreliable than China. There's been a near constant negative impact from the US doing something. At least with China, we are very good at winning trade wars with them here in Australia.
You might feel differently if you were a Filipino or Vietnamese fisherman whose family relied on the income from the stocks of the South China Sea, or a Uighur person living in Western China, or a Ukrainian soldier who has to deal with drones built with Chinese components, or a democracy advocate in Hong Kong, or arguably, a person who had plans for 2020-2021. Or, on a more local note, an Australian automotive worker…
> or a Ukrainian soldier who has to deal with drones built with Chinese components
man you're gonna be disappointed when you learn where the components for Ukrainian drones come from (spoiler alert, it's China 95% of Ukrainian drone manufacturers use Chinese components. Both Ukrainian and Russian drones are Chinese components glued together, the vendors in China literally stagger Russian and Ukrainian buyers on the factory floors to not have them run into each other). The largest trade partner of Vietnam and the Phillipines is China.
The kind of thinking that assumes that rivalry implies deglobalization or bloc politics is exactly what's 30 years out of date. It's projecting how Americans think on the entire world, but that's not how the world works any more. The rest of the world continues to globalize, even through war.
America is undergoing Sovietization and erecting an Iron Curtain, and China ironically enough is simply doing what the US used to do. If Americans think the rest of the world will follow them into isolation they're going to make the same discovery the Russians did in the last century.
Re: GLM 5.2 beats Claude in our benchmarks
#349[dead]
Re: GLM 5.2 beats Claude in our benchmarks
#350Earlier quoted context omitted.
> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…
What is your favorite harness for the open weights?
pi is great for learning, oh-my-pi has all the nice things included that I've built fory pi previously.