Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

261–270 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#261

Post by JumpCrisscross »

DennisP wrote: post #249
JumpCrisscross wrote: post #216

Earlier quoted context omitted.

> they just compress things down but they're still in there The compression is almost certainly in part specific knowledge getting fuzzed.

Yeah, but it's everything getting fuzzed, including the parts you care about.

Sure. There is a legitimate question around whether one can selectively excise “useless” knowledge. My guess is you can’t. The act of learning it encodes both the act of learning and the knowledge per se. The former is the power of the LLM. (I personally force mine to double check everything instead of going off memory.)

Re: GLM 5.2 beats Claude in our benchmarks

#262

Post by jfaat »

gertlabs wrote: post #172
pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

> but if you only want to use the best model available, it isn't there yet

I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way more stable. Almost like buying a Ferrari for your daily commute instead of a Toyota or even a Mercedes.

I think there are several factors. Certainly marketing making us think we need the shiny thing which is rampant online and very smart people think they aren't susceptible to. There's a lot of really odd 'I trust Anthropic/OpenAI more than Deepseek' which tends to ignore, for starters, that you can run choose your provider and still save a ton. I also think there's some amount of addiction and brand loyalty where a Ferrari is one hell of a drive so that you turn your nose up at that sensible Toyota. Oh the other one I see used is like oh only fable can oneshot updating my embedded systems thing from 1975 to rust which is great but let's recognize how niche that is.

And it ends up just coming across as people are getting SO reliant on the tools so fast. Maybe it's ok to think and like read a few lines of code and work with these agents to convert your thing to rust or center your div. Even if coding is over which in some sense it certainly is, don't turn your mind into the wall-e people yet. I found myself guilty of this so often. It takes way more time and effort to do things via prompt and I wouldn't just open the editor and fix it because that dopamine hit of the magic the abstraction provided was so strong.

So I'm pretty much done using the 'best' (on benchmarks, if money isn't an object, etc etc) models available. After a year on Sonnet/Opus/GPT5x I'm having way better results with open weights models that don't get lobotomized weekly. I'm finding ways to do the crafting part of building software by focusing on honing my harness and workflow. I'm enjoying changing the oil on my Toyota after a year of almost flying off cliffs in my Ferrari and if I can check my ego it's a purely positive thing.

Re: GLM 5.2 beats Claude in our benchmarks

#263

Post by botro »

tiahura wrote: post #164
Sanzig wrote: post #143

Earlier quoted context omitted.

Anyone done any benchmarks on the NV4FP quant? Seriously considering pitching an 8 x RTX 6000 Pro box at work to run GLM-5.2 in an air gapped environment.

Good luck. I’m in the legal field, and even there, selling airgapped is tough.

What are the challenges you've seen in selling air gapped? Is it the high upfront cost? Challenges with hardware maintenance or something else?

Re: GLM 5.2 beats Claude in our benchmarks

#264

Post by BobbyJo »

chillfox wrote: post #197
woeirua wrote: post #136

Earlier quoted context omitted.

You are delusional if you think China is going to let Europe have access to Mythos level models for free.

Why not? Mythos level really doesn't seem that scary. And it would be a great way to take away the American labs international market. I think it would make strategic sense for them to release more capable models than what American labs are allowed to make available to the world. It would help them grow their global soft-power and be a destabilizing effect on the American economy.

It is fairly obvious to me that the open models are a form of "dumping" as far as the economics and the desired outcome from China's perspective. They get to watch as the US pours tons of money and talent into an industry, then prevent that investment from having any return. In 5 years we'll be on equal footing, China will have spent 1/1000th the money, and the only downside will be that they spent 5 years being 6 months behind.

China could not be happier.

The same model is going to apply to the silicon supply chain as well is my guess. 1000th the expenditure in exchange for being a little behind the curve.

I worry it will have a very real chilling effect on research and development, since customers will probably very quickly switch to the thing that costs 1/10th as much, sucking out the ROI.

Re: GLM 5.2 beats Claude in our benchmarks

#266

Post by TimXare »

pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

[dead]

Re: GLM 5.2 beats Claude in our benchmarks

#267

Post by acters »

himata4113 wrote: post #14

These numbers are seem pretty low compared to what I was able to achieve specifically around windows kernel, win32k win32u to be exact. It honestly wouldn't surprise me anymore if china started surpassing models that US makes public, at least in specific categories such as cyber. GLM 5.2 is already capable enough to assist in self-training which is similar to what we saw happen with frontier models and they appear to…

I am finding Chinese models are introducing more guidelines against cyber. Especially Kimi k2.7 code seems to have extra training against cyber security capabilities. Last one, k2.6 was a lot stronger at cyber but obviously the Kimi team improved over time, so this is not the best they can do but no one will be able to get the best anymore.

I expect future Chinese models to introduce even more of this type of bogus "safety" training.

Looks like if you are a white hat, then you will be fighting an uphill battle. Black hats will be fine, they will not care, they can just run a heretic model or specialty trained model.

Re: GLM 5.2 beats Claude in our benchmarks

#268

Post by thefourthchime »

_s_a_m_ wrote: post #129

I tried GLM many times and it is bad, i have on clue what these people are talking about

Same. I asked it my Pac-Man question and it was the first to DNF.

It just goes off getting confused about how to design the map for 15 minutes and then times out.

Re: GLM 5.2 beats Claude in our benchmarks

#270

Post by pizzly »

echelon wrote: post #258
ronsor wrote: post #245

Earlier quoted context omitted.

Opus 4.6 is still my preferred model for work, so this is great to hear.

I can't wait for open models to take over in all categories. Sounds like this is the year for coding.

It looks possible open models will. I never expected the reason would be political/legal rather than technical.
Post reply on HN