[dead]
GLM 5.2 beats Claude in our benchmarks
511–520 of 559 posts
Re: GLM 5.2 beats Claude in our benchmarks
#512Earlier quoted context omitted.
GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…
I find it hard to trust a ranking system that gives Sonnet a higher capability score than Fable.
It would have made things easier for us if Sonnet 4.6 scored lower, but it's a great model and the data is real.
It doesn't have a higher capability score than Fable, though. We break our coding evaluations into 2 parts, and "one-shot coding" makes up part of the index, where Fable significantly outperforms every other model, which is why it's ranked at the top despite Sonnet 4.6 having a slightly higher median (and lower average) in long-horizon agentic workloads. One-shot coding tends to be the most correlated with other companies' model cards, whereas agentic coding is partly about how well a model can adapt to a custom harness. Fable also refused some tasks.
Data at https://gertlabs.com/rankings?ow=1&mode=oneshot_coding
Re: GLM 5.2 beats Claude in our benchmarks
#513Earlier quoted context omitted.
I think people are grouping into two flows. One group is trying to get the LLM to basically one shot everything and not properly reviewing the output. Others are using the LLM to assist their human intelligence in a tight loop. If you’re doing the former you really do need the best model available because that’s still right on the edge of what LLMs can do at best, and at worst you’re just shipping pure unmaintainable…
The later takes too much mental ressources, the same when reviewing truly the code generated by the former. I generally started by reviewing but after a while (maximum in hours), I just can't keep up and resort to LLMs as sole reviewers.
not many want to admit this
Re: GLM 5.2 beats Claude in our benchmarks
#514Earlier quoted context omitted.
"rote memorization" is not the right way to describe how an LLM works. The memorization of say 100000 world facts through training texts, which enrich model associations all around, is absolutely not the same as rote memorization on 10^50 digits of pi. Not for a human, and even more so, not for an LLM. An LLM trained with digits of pi and one trained with books and posts, even if they both have the exact same amount…
> actual human books vs possible permutations of characters and spaces You're making the assumption that anything produced by a human necessarily contains more useful information than random noise does. This is false. Even when only considering human intelligence, it's entirely possible to absorb information that makes you stupider, not smarter; learning is only valuable if you actually learn the right things.
[flagged]
Re: GLM 5.2 beats Claude in our benchmarks
#515Earlier quoted context omitted.
US EVs were also heavily subsidized, but they were all built using Chinese parts.
The EV supply chain in the US back in say 2007 certainly had far fewer key parts sourced from China than recent years. As far as US EVs being subsidized early, if you take state and federal tax incentives, DoE grants and loan guarantees as subsidizes then that's true. It's debatable (I think incentives applied to all suppliers not just US ones) but a reasonable statement.
Tesla given $60M by Obama admin when they were deep in debt and may have gone out of business.
so Tesla technically is subsidized by US govt. SpaceX too. Without NASA funding, they'd be long out of business.
China and US ain't that different.
China realizes that being a tech and industrial powerhouse working on future tech is great for their economy. They bet huge on it. That's how they win.
Europe on the other hand is now a laggard.
Re: GLM 5.2 beats Claude in our benchmarks
#516Earlier quoted context omitted.
> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…
I’m writing a lot of React code and find that the cheaper models are pretty terrible. Maybe I’m holding it wrong but the experience that the cheaper model is usually enough just track with my experience. Worse, I find predicting the difficulty of tasks exceedingly difficult. More often than not using the initially cheaper models requires me to reroll with a more expensive one or waste a lot of times and tokens cleani…
What happens when you find the models are terrible? The claimed results don't match? My dev cycle tends to be write a test for blah blah, add feature to satisfy test, make sure tests pass.
Re: GLM 5.2 beats Claude in our benchmarks
#517Earlier quoted context omitted.
Z.ai does not lock you in to any harness.
Is there a secure way to use GLM without spending $10K’s for local HW? I “only” have a 128GiB inference machine, and don’t really trust anthropic not to steal my IP over time. I see no reason to trust Z.ai more than other vendors.
Ollama Cloud has a $20 a month subscription. They say they retain 0 information. And rather than token based billing, it's GPU time billing.
Re: GLM 5.2 beats Claude in our benchmarks
#518There's no question to me, after trying both, that Fable is much better than GLM-5.2 when left alone in front of hard coding tasks
Now maybe what plateaus is the human collaboration efficiency, because at some point it will be bottlenecked by the human
Thus companies who still try to have humans perform intertwined work with their AI won't see an improvement, while the ones who fin the right conditions to give their AI more free rein will see it.
Kind of like it's no use having a workhorse pull a combine harvester : at some point, when machines reach sufficient efficiency, you just give wheels to the harvester and let it run.
Re: GLM 5.2 beats Claude in our benchmarks
#519Earlier quoted context omitted.
GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…
> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…
> I'm trying to wrap my head around exactly why so may people seem to want the best model available
To me this is a "more expectations mean more disappointment" situation.
Some people have higher expectations than others, and even the best model available is not good enough for what those people really want it to do once you start digging. In that light, the goal is not using the best model, but rather using the least insidiously deficient model.
Many people chase the edge because it's the least disappointing.
> when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price.
The fatuousness of this statement pretty quickly becomes apparent if you spend more time looking at it, IMO, because the venn diagram of "damn good" and "not nearly good enough" strongly overlaps. Even the best model writing excellent lines of code still has noticeably deficient ability to decide which excellent lines of code to write. The goal is to improve the separation between them, not save a few dollars, because the emotional effort is worth more to us than the money.
> And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way more stable.
Your minimization of performance differences and maximization of stability differences is exposing your biases.
Side note: I think you should know that to me at least some of what you said reads like self-rationalized moralizing. I couldn't help but imagine Principal Skinner saying "Am I so out of touch? No, it's the children who are wrong." People don't only want different things than you do because they don't know what they're doing.
Re: GLM 5.2 beats Claude in our benchmarks
#520Earlier quoted context omitted.
> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…
> most halfway decent models can write damn good code for a fraction of the price. The difference is how the model is used. With Opus you can give it a long-horizon task (eg build an entire feature) and it will plan it out and implement it and almost always stay on task. This is what people mean when they say "agentic tasks" With the lessor models the code is fine, but they need something else to plan what needs to b…
I don't know about you guys, but half of the time I give Opus something actually complicated, it spends 50+ minutes trying to understand the problem, running lots of searches and tool calls, and then gives up and just writes a brief summary of what it thought about. Biggest waste of tokens you can imagine.