Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

321–330 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#321

Post by danmaz74 »

woeirua wrote: post #136
danmaz74 wrote: post #103

Earlier quoted context omitted.

It will almost for sure surpass the models which Trump will allow US "allies" (which he just considers client states) to use. This, together with China's growing dominance in PV, rechargeable batteries, EV, could really be the nail in the coffin for the post WWII economic world order.

You are delusional if you think China is going to let Europe have access to Mythos level models for free.

We'll see. Helping Trump in destroying USA's traditional alliances is probably worth more to China than keeping a Mythos for themselves.

Re: GLM 5.2 beats Claude in our benchmarks

#322

Post by versteegen »

scottcha wrote: post #224
computerex wrote: post #163

Earlier quoted context omitted.

I have had bad experience with neuralwatt GLM 5.2. Seems like they may be using quantized version of the model.

Hi I'm the CTO of neuralwatt, would love to hear your feedback on what your experience was. Feel free to email me scott@neuralwatt.com. Also for GLM5.2 we run the FP8 quantization at 1M context which is a common deployment target.

Hi Scott! Was just considering signing up, NW looks great (fp8 GLM 5.2 is good!) Standard cached token pricing for GLM 5.2 is pretty high, I'm wondering whether the KV cache for that model actually is that expensive to serve on average, or if Neuralwatt's energy pricing for long-running GLM 5.2 agents is especially competitive? The live energy stats don't break down by token type, would love to see that. And 2/3 of the examples given in docs/energy-methodology are models you don't even host anymore. Uncertainty and selective stats puts people off signing up, they tend to assume the worst. Oh, and MiMo or DS4 please :)

Re: GLM 5.2 beats Claude in our benchmarks

#323

Post by pimeys »

wahnfrieden wrote: post #300
pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

Why are you spending on API for GPT coding instead of stacking 20x subs and using codex-lb?

Company pays API prices so we can use daily the best model for our job without being locked in. Also the team subscriptions started to be more like X per seat + usage...

Re: GLM 5.2 beats Claude in our benchmarks

#324

Post by johne20 »

scottcha wrote: post #224
computerex wrote: post #163

Earlier quoted context omitted.

I have had bad experience with neuralwatt GLM 5.2. Seems like they may be using quantized version of the model.

Hi I'm the CTO of neuralwatt, would love to hear your feedback on what your experience was. Feel free to email me scott@neuralwatt.com. Also for GLM5.2 we run the FP8 quantization at 1M context which is a common deployment target.

I had good experience with neuralwatt in my heavy testing on real project in last days. Price/performance for api pricing was great. When using with pi, I was a little confused on if/how it supports diff reasoning levels?

Re: GLM 5.2 beats Claude in our benchmarks

#325

Post by gguncth »

pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

What makes you use API billing instead of a plan?

Re: GLM 5.2 beats Claude in our benchmarks

#326

Post by pimeys »

Chrisoaks wrote: post #236
pimeys wrote: post #76

Earlier quoted context omitted.

API, pay per token.

Why are you not using the subscription plan?

I want to run the model in Western servers. And GPT/Opus is paid by the company which doesn't really get subsidized tokens.

In the future none of us do, so it's better to trial how the actually adorable models perform.

Re: GLM 5.2 beats Claude in our benchmarks

#327

Post by nl »

jfaat wrote: post #262
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

> most halfway decent models can write damn good code for a fraction of the price.

The difference is how the model is used.

With Opus you can give it a long-horizon task (eg build an entire feature) and it will plan it out and implement it and almost always stay on task. This is what people mean when they say "agentic tasks"

With the lessor models the code is fine, but they need something else to plan what needs to be done.

GLM-5.2 is the third model (after Opus 4.6+ and GPT-5.5) that can do this agentic style work.

Notably Gemini 3.1 Pro is notoriously bad at this style work - the code is good, but it drifts off task most of the time. 3.5 Flash is supposed to address this, but I haven't had a good reason to try it.

Re: GLM 5.2 beats Claude in our benchmarks

#328

Post by nl »

Madmallard wrote: post #222
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

Notice the website url is the same name as the commentor. Notice he's using "trust me bro" benchmarks. Can we just remove all the motivated speech on HN? This is just not trustworthy information at all and obviously is incentivized. Everyone is grinding and marketing nobody is actually discussing anything for real.

What does this even mean?

Re: GLM 5.2 beats Claude in our benchmarks

#329

Post by Aurornis »

InvertedRhodium wrote: post #226
cheschire wrote: post #192

Earlier quoted context omitted.

Not to mention the three separate dedicated 15A circuits you would need to have installed in order to run the 3x 2000W power supplies running ideally at no more than 1400W sustained load each. And definitely would need 200A service to the house if you have a family living there with you. But hey you could save on heating?

That’s a uniquely US issue - in NZ you can get a 100A single phase at 230V nominal without any issue. 23kw, straight to your door. A single circuit using 10mm TPS would technically be enough to run what you’re describing. Might be pricey though, I’d probably take the excuse to get 3 phase installed so I could get access to the stock of used 3 phase machinery.

> That’s a uniquely US issue - in NZ you can get a 100A single phase at 230V nominal without any issue. 23kw, straight to your door

In the US it's common to get 200A 120/240V split-phase service. We're talking about the wiring inside the house, though.

How do you think everyone here is charging their electric cars at home and running our AC and electric cooktops at the same time if we didn't also have that? :)

You need to derate for constant loads here, and I assume you have to do that in NZ as well.

So, no, not a "uniquely US issue".

Re: GLM 5.2 beats Claude in our benchmarks

#330

Post by SwellJoe »

onoesworkacct wrote: post #298
SwellJoe wrote: post #93

I added GLM 5.2 to my security bug hunting benchmark when it came out, and found it to be a good performer, but not the best open model. The benchmark tests whether models can find bugs Mythos found. The best open models in the initial benchmark were DeepSeek V4 Pro or MiMo 2.5 Pro. But it turned out MiMo got lucky, it's performed worse on almost every test I've done since, while DeepSeek has consistently been among…

could mimo have scraped the mythos findings already? it's very recent

That's covered in the article. All bugs (which you can see here: https://github.com/swelljoe/nelson/tree/main/cases ) are extremely recent (like a week old when I pulled them at the end of May). MiMo 2.5 Pro was released in April, at least a month before any of the cases were published, and I don't remember the exact training data cutoff for that one (if I found it), but I'm certain it's at least a couple/few months before the release date, as the base training when the data gets baked in is usually followed by weeks or months of post-training.

Anyway, it isn't possible for any of the models, so far, to be trained on the Mythos bugs. We're getting closer to the point where I have to worry about that, at which point I'll roll forward and pull some newer CVEs from what they've published, assuming they keep publishing new bugs. (And, if they don't, it's trivial to switch to just random CVEs. But, finding out what Mythos is up to is interesting.)

Post reply on HN