Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

331–340 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#331

Post by kingstnap »

Ldorigo wrote: post #180
dakolli wrote: post #55

Earlier quoted context omitted.

8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps.. Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of mon…

How do the economics of your statement work out? Clearly inference providers don't have a time to ROI of 10 years on their hardware costs; and that's without even taking ongoing energy costs into account. What's missing here?

Output tokens are actually kinda expensive for the provider.

The input cache hit tokens are incredibly cheap for them, (incredibly high margin too, except for deepseek).

And input tokens are in the middle. Input tokens can be processed very efficiently.

Also his math is wrong. $100k gets you 22.7B output tokens at $4.4/M which is how much GLM 5.2 costs.

At 500/s 22.7B is just 500 days. Or about 1.54 years. Which is much less then the life of the hardware.

Re: GLM 5.2 beats Claude in our benchmarks

#332

Post by SwellJoe »

amhoab wrote: post #278
SwellJoe wrote: post #93

I added GLM 5.2 to my security bug hunting benchmark when it came out, and found it to be a good performer, but not the best open model. The benchmark tests whether models can find bugs Mythos found. The best open models in the initial benchmark were DeepSeek V4 Pro or MiMo 2.5 Pro. But it turned out MiMo got lucky, it's performed worse on almost every test I've done since, while DeepSeek has consistently been among…

Aren't you the Webmin guy?

More the Virtualmin guy. But, yeah, I also work on Webmin and have since '99, so I'm a Webmin guy. But, Jamie is the Webmin guy. (And, I'll note that something like half of my commits to Webmin over the past few months have been bug fixes of bugs found by models, sometimes via Nelson, sometimes just interacting with Opus in Claude Code.)

Re: GLM 5.2 beats Claude in our benchmarks

#333

Post by NitpickLawyer »

nl wrote: post #327
jfaat wrote: post #262

Earlier quoted context omitted.

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

> most halfway decent models can write damn good code for a fraction of the price. The difference is how the model is used. With Opus you can give it a long-horizon task (eg build an entire feature) and it will plan it out and implement it and almost always stay on task. This is what people mean when they say "agentic tasks" With the lessor models the code is fine, but they need something else to plan what needs to b…

> 3.5 Flash is supposed to address this, but I haven't had a good reason to try it.

Price and speed, for me. GLM5.2 is "good enough" for some tasks, but rather slow (on their coding plan). In the time it takes GLM to "read files to figure out...", gemini flash is usually finished. It's not SotA for coding, but it's fast and often "good enough" for normal tasks.

Re: GLM 5.2 beats Claude in our benchmarks

#334

Post by SV_BubbleTime »

hedora wrote: post #297
SV_BubbleTime wrote: post #146

Earlier quoted context omitted.

Factory F350 Platinum is at least 90k sticker.

Yet Ford claims it is impossible to sell any pickups for > $60K, so they killed the lightning. I assume (since they claim they are selling the batteries to AI data centers), they’ll produce some sort of EV >= F150 once the bubble pops, and we get a new president.

Automotive EE here… every other decision about vehicles is about emissions. CAFE, the reason that a company releases X model is that they can then sell more Y models that get worse mileage.

EV is a separate thing. Vastly overmarketed for the technology as it exists today.

Re: GLM 5.2 beats Claude in our benchmarks

#335

Post by NitpickLawyer »

dakolli wrote: post #311
AussieWog93 wrote: post #219

Earlier quoted context omitted.

>Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. Not sure if you're being sarcastic, but I can run a quantised version of Gemma or Qwen on my 16GB M1 Macbook Pro that beats GPT-4 from 2023 hands-down. I wouldn't be surprised if, in another 3 years, you'd be able to run something as powerful as Opus 4.5 or GLM-5.2 on standard consumer hardware - s…

I'm being very sarcastic, local model evangalists seems to just be operating on vibes when they say these things and are completely disconnected from how models work, what the hardware requirements are. Prices aren't going down, and consumer platforms are being shipped with less RAM so we can be sold cloud products. This isn't going to happen. Can you please explain to me how you're going to fit 700bb-1T params in 64…

> Can you please explain to me how you're going to fit 700bb-1T params in 64GB of RAM?

You don't. What they're saying is that today's small models (that fit on consumer hw) are better than yesteryear's top models. GPT4 was reportedly 8x 220B (~1.6T) MoE, and today you can run a 30-120B model that beats it handedly in real-world tasks.

Similarly for 4-20B models beating GPT3 (175B) and so on.

There is a sweetspot of "good enough" that the small models can reach, where you get equivalent tasks solved fully locally. They'll never touch SotA, but they'll reach 2-3-4 year's SotA. Which, depending on the task you need, it can be "good enough".

Re: GLM 5.2 beats Claude in our benchmarks

#336

Post by robrenaud »

gertlabs wrote: post #172
pimeys wrote: post #59

I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars... This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and Open…

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

If a good SWE is $150/hour, does the model cost actually matter? Surely you'd be willing to spend $10/hour to make that SWE 20% more productive? The model cost is still much less than the salary.

Re: GLM 5.2 beats Claude in our benchmarks

#337

Post by mdp2021 »

tills13 wrote: post #70
WithinReason wrote: post #26

> [...] beating Claude Code (32%) at roughly $0.17 per vulnerability found Claude Code is an agent harness, not an LLM. Claude is a brand (or group of LLMs), not an LLM.

It costs nothing to not be pedantic.

"Kindly reach us in Cambridge for the lessons".

Re: GLM 5.2 beats Claude in our benchmarks

#338

Post by clhodapp »

cortesoft wrote: post #159
horsawlarway wrote: post #147

Earlier quoted context omitted.

My increasing frustration with these plans is the harness lock in. Anthropic won't even let you run "claude -p [prompt]" any more... They bill it at api rates. So if you're trying to automate the ai (and seriously, that's the point) the subsidized plans are crippled.

They postponed that change, here is the email they sent out: > In May, we sent you an email announcing that starting today, the Claude Agent SDK, claude -p, and third-party apps built on the Agent SDK would stop drawing from subscription rate limits and move to a dedicated monthly credit. We're writing to let you know that we’re not making this change today. We’re working to update the plan to better support how user…

Something I haven't been able to figure out.... How are you supposed to actually get an API key to use quota from your subscription? The terms of service still forbid using OAuth authentication and the API keys from the console indicate that you need to pre-load your account with funds when you try to use them.

Re: GLM 5.2 beats Claude in our benchmarks

#340

Post by davedx »

gertlabs wrote: post #283
neya wrote: post #276

Earlier quoted context omitted.

What is the methodology of your benchmark? On the contrary, I personally think these broader benchmarks are meaningless. I think personalized benchmarks are the way to go. They should answer "How does this model perform for MY use-case?" rather than trying to answer "How does this model perform across all coding environments?" Case in point: I use Elixir which is not as popular as Python, is always a hit or miss with…

We use a rotating pool of ~100 games for the coding parts of the benchmark, and are scored objectively based on ratings similar to Elo. Models write code submissions to interact with the environment, then are evaluated in large batches against other submissions. We test 11 popular/interesting languages (you can see the Languages chart to filter), but not Elixir -- although other evaluations have found that many LLMs…

An expressive and well designed language (elixir) is objectively better than a less well designed language like python. Python probably needs more LoC than elixir for the same task. Python is also untyped by default.
Post reply on HN