Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

381–390 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#381

Post by marci »

skeptic_ai wrote: post #205
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

Why Deepseek v4 flash is better than pro in your benchmarks?

This was a preview release. They haven't finish training. The Pro contains more knowledge but it probably takes longer training than flash for the smarts to kick in.

Re: GLM 5.2 beats Claude in our benchmarks

#382

Post by kennywinker »

rolisz wrote: post #342
robrenaud wrote: post #336

Earlier quoted context omitted.

If a good SWE is $150/hour, does the model cost actually matter? Surely you'd be willing to spend $10/hour to make that SWE 20% more productive? The model cost is still much less than the salary.

With Claude Code Ultrathink, I used 3 million tokens in 20 minutes. At API prices, that would be around 30$. So 90$/h. Model cost is not that much lower.

x40hrs/week * 50 weeks = $180k

Congrats, now you’re paying an engineer’s salary to make your engineer at best 20% more productive.

Better to hire another engineer, or two jrs, and build up your in house talent.

Re: GLM 5.2 beats Claude in our benchmarks

#384

Post by huksley »

horsawlarway wrote: post #147
Aditya_Garg wrote: post #139

Earlier quoted context omitted.

Im really curious about this. Why pay API pricing? I burn 1000s of dollars a month of api according to claude usage but only pay the $100 subscription

My increasing frustration with these plans is the harness lock in. Anthropic won't even let you run "claude -p [prompt]" any more... They bill it at api rates. So if you're trying to automate the ai (and seriously, that's the point) the subsidized plans are crippled.

They reverted this decision, "claude -p [prompt]" works with your subscription ok.

Re: GLM 5.2 beats Claude in our benchmarks

#385

Post by Mashimo »

maxk42 wrote: post #196
TurdF3rguson wrote: post #105

Earlier quoted context omitted.

It also means giving up vision which I don't know how I would deal with. I think I would prefer a weaker model with vision than a stronger without.

Openrouter definitely supports vision models. Why would you have to give up vision?

> Why would you have to give up vision?

Because you would have to switch model.

You can't just say "Oh, button X looks weird see [screenshot]" while coding with GLM. You would need to switch to another model and then maybe back.

Re: GLM 5.2 beats Claude in our benchmarks

#386

Post by dsrtslnd23 »

jfaat wrote: post #262
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

I agree, but there are use cases for the 'best model' other than converting your 1975 stuff to rust: for use cases where LLMs are just getting started to be useful I really want to use the current 'best' model: e.g. CAD, PCB design etc. In particular anything which requires spatial reasoning. The short time I had access to Fable 5 - it was just way better than any other model.

Re: GLM 5.2 beats Claude in our benchmarks

#387

Post by Kiog-Aser »

uluckydev wrote: post #309

I used Claude a lot, but with Claude Code it takes a lot of context window, and it's very pricey, to be honest. Then I shifted towards Minimax. I used the coding plan because it's cheaper, but it still gets the job done. When M3 came out, I started using it, and it was actually really good. After that, I shifted towards OpenCode for my AI agent, and that's been really good as well. The best thing I realized is that i…

[dead]

Re: GLM 5.2 beats Claude in our benchmarks

#388

Post by throwawayffffas »

hedora wrote: post #288
throwawayffffas wrote: post #169

Earlier quoted context omitted.

Z.ai does not lock you in to any harness.

Is there a secure way to use GLM without spending $10K’s for local HW? I “only” have a 128GiB inference machine, and don’t really trust anthropic not to steal my IP over time. I see no reason to trust Z.ai more than other vendors.

Kind of, you need at least 256 gb of vram and 24-40 gb of vram to run the 2bit quantization, because it's a moe you just need the expert to fit in vram to get significant improvement over a pure CPU setup. At 2bits though expect significant quality loss.

Re: GLM 5.2 beats Claude in our benchmarks

#389

Post by peheje »

jfaat wrote: post #262
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

Reason people want the best: people want to believe their project is so advanced that they need the most clever LLM possible. To say otherwise is to admit that it's not really frontier or novel in any way. And people don't like that.

Re: GLM 5.2 beats Claude in our benchmarks

#390

Post by swiftcoder »

nl wrote: post #327
jfaat wrote: post #262

Earlier quoted context omitted.

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

> most halfway decent models can write damn good code for a fraction of the price. The difference is how the model is used. With Opus you can give it a long-horizon task (eg build an entire feature) and it will plan it out and implement it and almost always stay on task. This is what people mean when they say "agentic tasks" With the lessor models the code is fine, but they need something else to plan what needs to b…

> With Opus you can give it a long-horizon task (eg build an entire feature) and it will plan it out and implement it and almost always stay on task. This is what people mean when they say "agentic tasks"

I've had no trouble getting the current generation of smaller models to do the same thing. Maybe it's more of a harness issue than a model issue?

Recently I've used both MiniMax M3 and DeepSeek V4 Flash to one-shot moderately complex applications from a written spec, and neither one got lost along the way

Post reply on HN