Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

361–370 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#361

Post by YmiYugy »

robrenaud wrote: post #336
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

If a good SWE is $150/hour, does the model cost actually matter? Surely you'd be willing to spend $10/hour to make that SWE 20% more productive? The model cost is still much less than the salary.

But SOTA models used liberally at API pricing is a lot more than $10/hour. You can probably burn $100+/hour with just a single agent, and probably thousands when running agents programmatically, e.g. workflows.

Re: GLM 5.2 beats Claude in our benchmarks

#362

Post by pimeys »

OtherShrezzing wrote: post #347
robrenaud wrote: post #336

Earlier quoted context omitted.

If a good SWE is $150/hour, does the model cost actually matter? Surely you'd be willing to spend $10/hour to make that SWE 20% more productive? The model cost is still much less than the salary.

I don’t think any engineers who cost $150/hr are having their productivity moved by 20% depending on a $10/hr gap between models on or near the frontier. Most of the gains right now come from tooling and process and any big post 2025 language model. The specific model isn’t that important right now.

Exactly. And being able to choose your own tools is much more valuable than having a tiny bit better model.

Re: GLM 5.2 beats Claude in our benchmarks

#363

Post by wahnfrieden »

pimeys wrote: post #323
wahnfrieden wrote: post #300

Earlier quoted context omitted.

Why are you spending on API for GPT coding instead of stacking 20x subs and using codex-lb?

Company pays API prices so we can use daily the best model for our job without being locked in. Also the team subscriptions started to be more like X per seat + usage...

Oh it sounded like personal use.

I understand the reasons to use team/enterprise accounts, but apart from the policy/management/billing side of it, I still don't understand the value in spending thousands for API instead of hundreds - even when there's argument that one provider is better than another depending on the use case, I don't think that credibly extends much beyond OpenAI + Anthropic frontiers, which both have $200 subs you can stack.

Re: GLM 5.2 beats Claude in our benchmarks

#364

Post by kelnos »

Title is misleading (and is editorialized from the actual article title). GLM 5.2 did better than Claude in one specific cybersecurity-related benchmark (finding vulnerabilities of one certain type). I don't think you can draw any general conclusions about the relative utility of the two models.

Re: GLM 5.2 beats Claude in our benchmarks

#367

Post by ifwinterco »

jfaat wrote: post #262
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

I think people are grouping into two flows.

One group is trying to get the LLM to basically one shot everything and not properly reviewing the output.

Others are using the LLM to assist their human intelligence in a tight loop.

If you’re doing the former you really do need the best model available because that’s still right on the edge of what LLMs can do at best, and at worst you’re just shipping pure unmaintainable slop.

If you’re doing the latter then you can get away with a slightly less powerful model without it making a material difference because your human intelligence is filling in gaps

Re: GLM 5.2 beats Claude in our benchmarks

#368

Post by 40four »

WinstonSmith84 wrote: post #357
40four wrote: post #351

It’s hard to argue against the open weight models if your only concern is coding. Which, for many of us hackers here in this forum, it is. But I would like to point out that the overwhelming majority of people using LLMs aren’t programmers, don’t care about coding, and couldn’t even be bothered to “vibe code”. So we should consider the bias of the output of these open weight models, and what that looks like, outside…

There is no money made from these people though .. people who are using ChatGPT to plan for their next week-end or their next vacation aren't paying a $100 or $200 monthly subscription. As for non coder office workers (accountants, PMs, etc.), they use Microsoft or Google products which all integrate AI to some extent within their products - with RAG for Sharepoint to some basic AIs to generate text or automate work…

You’re making a good point. I don’t disagree with what you’re saying. But I think my point got lost.

I don’t agree with “Software development is where money is made for these labs”. Coders will inevitably eat up the most tokens & buy the bigger $200 subscriptions because we want to keep working.

But us coders are still the small minority of users. They aren’t counting on us to get to trillion dollar evaluations.

They are counting on the regular folks to buy the $20/ month subscription. It’s really easy to run out your free tier usage these days, asking questions that have nothing to do with coding.

So my point is what does that output look like for someone asking a question about politics or world news?

Re: GLM 5.2 beats Claude in our benchmarks

#369

Post by childintime »

About running models locally and why data centers win (for now): they can stream the model weights to many neural engines at the same time, so each of these only needs enough RAM to hold the KV cache. So each engine is cheaper to operate, plus they are time-shared, resulting in massive wins for data centers.

So one can see businesses owning their own such cluster, next to their database infra, in the near future.

Post reply on HN