Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 beats Claude in our benchmarks

semgrep.dev

Post reply on HN
Print view View original

311–320 of 559 posts

Re: GLM 5.2 beats Claude in our benchmarks

#311

Post by dakolli »

AussieWog93 wrote: post #219
dakolli wrote: post #55

Earlier quoted context omitted.

8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps.. Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of mon…

>Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. Not sure if you're being sarcastic, but I can run a quantised version of Gemma or Qwen on my 16GB M1 Macbook Pro that beats GPT-4 from 2023 hands-down. I wouldn't be surprised if, in another 3 years, you'd be able to run something as powerful as Opus 4.5 or GLM-5.2 on standard consumer hardware - s…

I'm being very sarcastic, local model evangalists seems to just be operating on vibes when they say these things and are completely disconnected from how models work, what the hardware requirements are.

Prices aren't going down, and consumer platforms are being shipped with less RAM so we can be sold cloud products. This isn't going to happen.

Can you please explain to me how you're going to fit 700bb-1T params in 64GB of RAM? You realize there are memory requirements proportional to model size?

Re: GLM 5.2 beats Claude in our benchmarks

#312

Post by marcus_holmes »

byzantinegene wrote: post #302
marcus_holmes wrote: post #235

Earlier quoted context omitted.

This is clearly where the industry is going, imho. Everyone who is playing with LLMs wants a laptop with enough grunt to run a decent model locally. We've been sat with basically the same PC specs for ~20 years - our current specs are within an order of magnitude of the ones we could buy back in 2010. This is not really constrained by tech, as we could have much, much, larger machines. It's more because there's no ma…

my only concern if the same specs today would cost 10x more given the trajectory of the growth of memory prices lately.

I think this is where the new technology comes in. There is demand for 10x (or 1000x) the memory that we're using at the moment, so someone/something will satisfy that demand. We haven't had that demand up until now, because 16Gb was a perfectly reasonable amount of memory that could run pretty much anything, and if that won't then 32Gb will. There was zero demand for 16Tb memory machines because no-one had any application for that much memory. Now that's changing, and there is demand for that much, so we'd expect to see that being made available.

But the existing tech we're using for 16Gb probably isn't going to scale to 16Tb at a reasonable price point. And the price point is relatively inelastic - people are used to paying So there's room for a new technology to come in, where there wasn't previously. This is what happened all through the 90's, and we churned through a bunch of standards and technologies to try and keep up with demand.

Re: GLM 5.2 beats Claude in our benchmarks

#313

Post by dakolli »

Ldorigo wrote: post #180
dakolli wrote: post #55

Earlier quoted context omitted.

8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps.. Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years. For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of mon…

How do the economics of your statement work out? Clearly inference providers don't have a time to ROI of 10 years on their hardware costs; and that's without even taking ongoing energy costs into account. What's missing here?

https://aimultiple.com/gpu-benchmark

concurrency

Re: GLM 5.2 beats Claude in our benchmarks

#314

Post by hedora »

gertlabs wrote: post #307
hedora wrote: post #285

Earlier quoted context omitted.

In your box plots, 4.6 sonnet wins over all (even opus 4.6, the 4.8’s and fable). That’s not super surprising to me, but, given the apparent randomness of the stack ranking, is GLM actually worse than any of the Anthropic models? This looks like a 10-way tie to me.

We've spent some time trying to understand this anomaly, even re-running Sonnet 4.6 through our evaluations to see if that would bring down its scores... and it didn't. I don't know what they did differently, but it's basically Opus 4.6 with more temperature variability (some great responses, some less great, with an approximately frontier median response in agentic work specifically). It is smart, methodical and exc…

That tracks with my experience.

4.7 was so bad, I locked a bunch of my machines to 4.6.

I haven’t bothered locking the 4.8 machines to 4.6. There was a HN thread a while back where they run swe bench a few times a day and measure success rate and latency. It showed opus getting significantly dumber for the week before a recent launch.

It wouldn’t surprise me if they’re quantizing to improve margins or to hype models in comparative testing in order to defraud investors at IPO.

Or, maybe QA is hard. Anyway, I think they hit a performance wall sometime at or before 4.6.

Re: GLM 5.2 beats Claude in our benchmarks

#315

Post by cik »

jfaat wrote: post #262
gertlabs wrote: post #172

Earlier quoted context omitted.

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing. In multi-agent coding environments, GLM 5.2 i…

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

I've landed in a similar place by reducing effort and cutting up tasks. I find that more exacting specifications to the models, yield significantly less need for "effort". Combining each with multjple git worktrees and an integration branch for the current worktrees themselves has yielded incresible results.

This also allows me to play with, and mix codex, claude cli, and others. This is my happy spot for the last two months.

Re: GLM 5.2 beats Claude in our benchmarks

#316

Post by realusername »

andai wrote: post #290
jfaat wrote: post #262

Earlier quoted context omitted.

> but if you only want to use the best model available, it isn't there yet I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way mo…

Yeah, the funniest thing about everyone freaking out about Fable's capabilities recently was that for most of the stuff they were amazed by, you could get roughly the same result from DeepSeek Flash. I used to be obsessed with what's the best model. Then a while back when the new best model came out, I tested it on a task. I also tested its little brother (much smaller model from same company). They both completed th…

> They both completed the task perfectly except the "best" model (the bigger one) cost 5x more and took 3x longer...

Same for me, I certainly don't have the same definition of success and failure either.

A more expensive model has *less* rooms for wandering around than a cheaper model.

If Claude wanders around during 10min until finding the most obvious solution, then I count it as a failure.

Re: GLM 5.2 beats Claude in our benchmarks

#317

Post by Danox »

lenerdenator wrote: post #209
chillfox wrote: post #201

Earlier quoted context omitted.

So... how's that any different from using American stuff for those of us in the rest of the world? Over the last decade, the US has been way more unreliable than China. There's been a near constant negative impact from the US doing something. At least with China, we are very good at winning trade wars with them here in Australia.

You might feel differently if you were a Filipino or Vietnamese fisherman whose family relied on the income from the stocks of the South China Sea, or a Uighur person living in Western China, or a Ukrainian soldier who has to deal with drones built with Chinese components, or a democracy advocate in Hong Kong, or arguably, a person who had plans for 2020-2021. Or, on a more local note, an Australian automotive worker…

More what aboutism American Indians, Aborigines, Māori, Sami, New Caledonia, the Kanak people what do they all have in common? Sent to re-education camps at some point in time, some of them sterilized, And all treated his second class citizens. One of the reasons most countries are relatively quiet about the Chinese is that so many other countries have indigenous people that were treated pretty much the same at some point in time in their history…

Stop pretending there’s some type of moral high ground there isn’t. Disgusting.

Re: GLM 5.2 beats Claude in our benchmarks

#319

Post by KronisLV »

sidcool wrote: post #274

Genuinely curious. Say GLM 5.2 is better than Opus. But how does one go about using it by themselves?

The simplest would be either OpenRouter: https://openrouter.ai/z-ai/glm-5.2

Or grabbing their GLM Coding Plan directly: https://z.ai/subscribe

I went with the second one to try it out, feels pretty okay (with OpenCode, though Claude Code would also work), however it feels like I reach the weekly limits somewhat fast with their 65 USD Pro subscription. They also have that whole peak times thing going on and apparently it will get worse after September:

> Supported models and Visual Understanding MCP share the same usage quota. GLM-5.2 and GLM-5-Turbo consume quota at 3x during peak hours and 2x during off-peak hours. Limited-time benefit: off-peak usage is currently charged at only 1x quota through the end of September. Peak hours: 14:00–18:00 daily (UTC+8).

Re: GLM 5.2 beats Claude in our benchmarks

#320

Post by KronisLV »

TurdF3rguson wrote: post #105
cmrdporcupine wrote: post #100

I like GLM 5.2... ish. It's ok. I'd be mostly fine switching to it. I just can't find a cost effective way to do that. z.AI's coding plan is both overpriced and unreliable. ollama's is also overpriced. Paying by the token for it on openrouter etc is more expensive than just having a Codex or Claude coding plan. If you have to pay by the token, it's clearly cheaper. It's not competitive with a coding plan though.

It also means giving up vision which I don't know how I would deal with. I think I would prefer a weaker model with vision than a stronger without.

It's odd that the model doesn't support it directly, but they at least have https://docs.z.ai/devpack/mcp/vision-mcp-server
Post reply on HN