Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

211–220 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#211

Post by lebovic »

SwellJoe wrote: post #93

I added GLM 5.2 to my security bug hunting benchmark when it came out, and found it to be a good performer, but not the best open model. The benchmark tests whether models can find bugs Mythos found. The best open models in the initial benchmark were DeepSeek V4 Pro or MiMo 2.5 Pro. But it turned out MiMo got lucky, it's performed worse on almost every test I've done since, while DeepSeek has consistently been among…

GLM 5.2 and DeepSeek v4 Pro seem to approach security research differently. This benchmark was with GLM 5.1, but the patterns are similar: https://dualuse.dev/posts/deepseek-v4-thinks-different

Overall, I still think GLM 5.2 is the much stronger performer. It's hard to tell the difference between GLM 5.2 and Opus at <120k tokens.

Re: GLM 5.2 beats Claude in our benchmarks

#212

Post by ygjb »

dom96 wrote: post #170
pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

Twenty dollars? How are you comfortable spending that much to write something as simple as a matrix bot? Are people doing this kind of thing just super rich or am I missing something?

It's pretty simple. There are things that I do because it's fun, like gamedev. I hand code that, and don't use LLM tools because I like learning and building. I do lots of utility stuff coding for my wife's business, most of that is stuff I could do in a few hours. It's worth $20 to not spend a few hours doing it. It's a cost benefit tradeoff. I won't learn much fixing WordPress themes or adding a feature to her web page, or setting up an automation for her, so I don't see the point of doing that.

Same thing for stuff at work. Oh, the tables/schema changed and my queries broke? I could dork around with spark and cypher for an hour, or I can tell claude to update the queries for the new schema. At the rate I am paid, spending on Claude tokens is generally a better use of my resources.

Building a net new solution? Coding tools take a back seat until I get the core logic right, then I let automation handle web page and UI scaffolding.

Re: GLM 5.2 beats Claude in our benchmarks

#213

Post by [deleted] »

throwup238 wrote: post #43
manquer wrote: post #8

Earlier quoted context omitted.

While unlikely , it is not without precedent , there are restrictions on ASML a Dutch company to sell EUV machines

That’s because the Department of Energy originally funded and contributed IP to the EUV Corp joint venture between several semiconductor companies (including ASML and Intel). Their ability to export control EUV was part of that original agreement that the entire technology is built on.

[deleted]

Re: GLM 5.2 beats Claude in our benchmarks

#214

Post by solenoid0937 »

techpression wrote: post #56
Gigachad wrote: post #51

Earlier quoted context omitted.

Turns out toy drones are more useful in war than multi million dollar planes anyway.

Reaper and Predator are both drones and there’s really no comparison to toy drones in terms of sheer destruction and capabilities in general, the comparison is actually quite apt imo.

You're right. Toy drones have proven vastly more effective IRL.

The others are a waste of taxpayer money. Extraordinarily low return on investment (kill-on-investment?)

Re: GLM 5.2 beats Claude in our benchmarks

#215

Post by solenoid0937 »

dakolli wrote: post #57
solenoid0937 wrote: post #3

GLM export controls incoming? I predict Commerce will force OpenRouter, HuggingFace to take some open models down within the next few months. Not that it would make any sense.

Cool then everyone will just change their config to route through a provider overseas for an added 50-100ms latency. Who cares.

Countries and businesses that don't want to be sanctioned by the US government or the US financial system care - so all western countries and corporations.

Re: GLM 5.2 beats Claude in our benchmarks

#216

Post by JumpCrisscross »

KetoManx64 wrote: post #112
JumpCrisscross wrote: post #106

Earlier quoted context omitted.

> I do not need the whole model. I don't need the model to have knowledge of the rain history of Algeria nor how many colors are in the Russian flag Isn’t the performance gap between quantized and full models indicative that even if you aren’t using it directly, the model knowing the colors in the Russian flag does have something to do with the intelligence you demand?

Do quantized models specifically prune out specific knowledge? I think they just compress things down but they're still in there. You'd most likely need to do that when you're doing the initial model training, but I'm not expert.

> they just compress things down but they're still in there

The compression is almost certainly in part specific knowledge getting fuzzed.

Re: GLM 5.2 beats Claude in our benchmarks

#217

Post by JumpCrisscross »

kibwen wrote: post #127
JumpCrisscross wrote: post #106

Earlier quoted context omitted.

> I do not need the whole model. I don't need the model to have knowledge of the rain history of Algeria nor how many colors are in the Russian flag Isn’t the performance gap between quantized and full models indicative that even if you aren’t using it directly, the model knowing the colors in the Russian flag does have something to do with the intelligence you demand?

Quantizing is one thing. But in general it's self-evident that training the model on information that is irrelevant to your use case does not necessarily improve ability, otherwise you'd have AGI just from reinforcing your model on memorizing the first 10^50 digits of pi. Likewise, LLMs do not violate the laws of information theory, and therefore the only way to encode X amount of information in Y amount of bits wher…

> it's self-evident that training the model on information that is irrelevant to your use case does not necessarily improve ability

We don’t understand AI or natural intelligence well enough to make such statements. As for self evidence, cross-domain competence in humans and the rise of generalist models over domain-specific ones (on competence, not cost) seems to pretty directly tank your hypothesis.

Re: GLM 5.2 beats Claude in our benchmarks

#218

Post by rockwotj »

skeptic_ai wrote: post #205
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

Why Deepseek v4 flash is better than pro in your benchmarks?

I have also found deepseek flash beat pro in some of my own internal evals for tasklet.ai it’s really surprising and I don’t understand it

Re: GLM 5.2 beats Claude in our benchmarks

#219

Post by AussieWog93 »

dakolli wrote: post #55
bArray wrote: post #37

Apparently GLM 5.2 is 753B parameters [1], what kind of hardware are people using to run this locally? [1] https://huggingface.co/zai-org/GLM-5.2

8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps.. Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of mon…

>Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years.

Not sure if you're being sarcastic, but I can run a quantised version of Gemma or Qwen on my 16GB M1 Macbook Pro that beats GPT-4 from 2023 hands-down.

I wouldn't be surprised if, in another 3 years, you'd be able to run something as powerful as Opus 4.5 or GLM-5.2 on standard consumer hardware - say a 32GB/64GB M7 Pro.

I also wouldn't be surprised if, 3 years after that, cheaper hardware and improved model efficiency means that there's a much smaller gap between what you can run on a consumer CPU (which, with memory prices coming down, could look like a 256GB M9 or M10 Pro) and $100k GPU cluster.

Re: GLM 5.2 beats Claude in our benchmarks

#220

Post by crazylogger »

rvz wrote: post #210

Many people here are now realizing that open weight models are now able to compete against frontier closed models. This is where we are heading and why many closed labs are terrified of this affecting their bottom line and the reason why they want them banned from being released.

Actually they don't even need to compete against frontier closed models, they just need to work.

99.99% people's day jobs aren't competing for the Fields Medal or even finding security vulnerabilities. So it appears while TAM (total addressable market) of AI in general is huge, TAM for frontier LLMs is tiny. Efficiency gains at roughly the same performance might be all people care about from now on.

Post reply on HN