Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

301–310 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#301

Post by hedora »

woeirua wrote: post #136
danmaz74 wrote: post #103

Earlier quoted context omitted.

It will almost for sure surpass the models which Trump will allow US "allies" (which he just considers client states) to use. This, together with China's growing dominance in PV, rechargeable batteries, EV, could really be the nail in the coffin for the post WWII economic world order.

You are delusional if you think China is going to let Europe have access to Mythos level models for free.

Didn’t they already? Mythos isn’t even SOTA according to Anthropic (they point at GPT 5.5), and third party benchmarks have massive error bars where Fable, GPT 5.5 and GLM 5.2 overlap.

Re: GLM 5.2 beats Claude in our benchmarks

#302

Post by byzantinegene »

marcus_holmes wrote: post #235
AussieWog93 wrote: post #219

Earlier quoted context omitted.

>Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. Not sure if you're being sarcastic, but I can run a quantised version of Gemma or Qwen on my 16GB M1 Macbook Pro that beats GPT-4 from 2023 hands-down. I wouldn't be surprised if, in another 3 years, you'd be able to run something as powerful as Opus 4.5 or GLM-5.2 on standard consumer hardware - s…

This is clearly where the industry is going, imho. Everyone who is playing with LLMs wants a laptop with enough grunt to run a decent model locally. We've been sat with basically the same PC specs for ~20 years - our current specs are within an order of magnitude of the ones we could buy back in 2010. This is not really constrained by tech, as we could have much, much, larger machines. It's more because there's no ma…

my only concern if the same specs today would cost 10x more given the trajectory of the growth of memory prices lately.

Re: GLM 5.2 beats Claude in our benchmarks

#303

Post by hedora »

rgbrenner wrote: post #19
solenoid0937 wrote: post #3

GLM export controls incoming? I predict Commerce will force OpenRouter, HuggingFace to take some open models down within the next few months. Not that it would make any sense.

If that happens it'll be an absolute disaster. Imagine a scenario where Anthropic and OpenAI prohibit most US companies from using their latest models because of safety.. And meanwhile attackers use equivalent open source models to attack US companies. Any prohibition on open source models will do nothing to fix the problem.. since attackers will never feel bound to the law. All advanced models must be available for…

OpenAI and Anthropic are already unable to make SOTA models generally available (and support this, oddly enough).

If huggingface or whatever is forced to take down open source licensed weights, there’s always bittorrent.

Export controls are one thing, but the US doesn’t really have import controls, and there’s no copyright issue, so DMCA, etc don’t come into play.

It’d take the courts years to decide how to contort the law to ban open weight models, and by then, it’ll be too late (and also pointless).

Re: GLM 5.2 beats Claude in our benchmarks

#304

Post by avereveard »

jchw wrote: post #186
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

After having used GLM 5.2 and Opus 4.8 for enough time, I'm very unconvinced of the benchmark maxxing claims - if anything, GLM 5.2's rather lackluster performance on benchmarks compared to Opus 4.8 paints the opposite picture when compared to the subjective experience. When I first used Opus 4.8, I threw several different workloads I had at it - I have Claude doing a lot of misc projects whose primary purpose is pre…

I really dislike opus 4.8 it rarely compete things and prefer to waste tokens making lists of things that are missing. When stuck or need input it words the challenge at length without conveying anything useful for decision making, and quite often its solution to problems is to excise features or just try catch errors and proceed with faulty data silently

Re: GLM 5.2 beats Claude in our benchmarks

#305

Post by hedora »

aussiegreenie wrote: post #45
solenoid0937 wrote: post #3

GLM export controls incoming? I predict Commerce will force OpenRouter, HuggingFace to take some open models down within the next few months. Not that it would make any sense.

The Americans may ban the use of the Chinese models in America. But like the Chinese car ban, everyone else will use them.

Technically speaking, Chinese cars have not been banned. They are subject to a 100% tariff. They’d still be price competitive, but the manufacturers haven’t bothered jumping through the regulatory hoops.

I’ll happily pay a 100% tariff on open weight models, and there are no regulatory hurdles for them to jump through (yet).

Re: GLM 5.2 beats Claude in our benchmarks

#306

Post by qingcharles »

SwellJoe wrote: post #93

I added GLM 5.2 to my security bug hunting benchmark when it came out, and found it to be a good performer, but not the best open model. The benchmark tests whether models can find bugs Mythos found. The best open models in the initial benchmark were DeepSeek V4 Pro or MiMo 2.5 Pro. But it turned out MiMo got lucky, it's performed worse on almost every test I've done since, while DeepSeek has consistently been among…

Every time a new frontier model arrives I have it give one specific codebase of mine a once-over for bugs and other idiotic mistakes.

Fable found a couple of good ones, then we lost Fable, so I tried GLM5.2 and it found two critical bugs that Fable had missed, so it got my seal of approval.

Re: GLM 5.2 beats Claude in our benchmarks

#307

Post by gertlabs »

hedora wrote: post #285
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

In your box plots, 4.6 sonnet wins over all (even opus 4.6, the 4.8’s and fable). That’s not super surprising to me, but, given the apparent randomness of the stack ranking, is GLM actually worse than any of the Anthropic models? This looks like a 10-way tie to me.

We've spent some time trying to understand this anomaly, even re-running Sonnet 4.6 through our evaluations to see if that would bring down its scores... and it didn't. I don't know what they did differently, but it's basically Opus 4.6 with more temperature variability (some great responses, some less great, with an approximately frontier median response in agentic work specifically). It is smart, methodical and excellent at tool calling in our custom environments.

We now use Sonnet 4.6 for a number of internal use cases we wouldn't have considered otherwise.

Re: GLM 5.2 beats Claude in our benchmarks

#308

Post by neya »

gertlabs wrote: post #283
neya wrote: post #276

Earlier quoted context omitted.

What is the methodology of your benchmark? On the contrary, I personally think these broader benchmarks are meaningless. I think personalized benchmarks are the way to go. They should answer "How does this model perform for MY use-case?" rather than trying to answer "How does this model perform across all coding environments?" Case in point: I use Elixir which is not as popular as Python, is always a hit or miss with…

We use a rotating pool of ~100 games for the coding parts of the benchmark, and are scored objectively based on ratings similar to Elo. Models write code submissions to interact with the environment, then are evaluated in large batches against other submissions. We test 11 popular/interesting languages (you can see the Languages chart to filter), but not Elixir -- although other evaluations have found that many LLMs…

Thanks!

Re: GLM 5.2 beats Claude in our benchmarks

#309

Post by uluckydev »

I used Claude a lot, but with Claude Code it takes a lot of context window, and it's very pricey, to be honest. Then I shifted towards Minimax. I used the coding plan because it's cheaper, but it still gets the job done. When M3 came out, I started using it, and it was actually really good. After that, I shifted towards OpenCode for my AI agent, and that's been really good as well. The best thing I realized is that it uses less context, works better, and gives me access to a lot of different models from one place. I never actually used GLM, but I recently found QuanCode, which is amazing. I used it to build a full-stack application. Now I'm shifting my focus more toward SaaS distribution. I'm still figuring out how to automate different workflows, and using QuanCode has been really fast and effective for building those automations.

Re: GLM 5.2 beats Claude in our benchmarks

#310

Post by ComplexSystems »

gertlabs wrote: post #172
pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

Sonnet 4.6 is ahead of Opus 4.7? Hm.
Post reply on HN