Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 vs. Opus

techstackups.com

Post reply on HN
Print view View original

281–290 of 367 posts

Re: GLM 5.2 vs. Opus

#281

Post by ramraj07 »

jameswhitford wrote: post #4
meander_water wrote: post #2

> So we ran it head-to-head against Claude Opus 4.8: same one-shot prompt, build a 3D platformer in raw WebGL from scratch Running a single one-shot prompt is not a benchmark, not is it representative of any sort of real-world usage. Most agent usage is collaborative so you need to test things like reliability (when I delegate a task, does it complete it without making up test results for e.g.) and steerability (does…

Hi, I am the author, I completely agree! I set out to run a vibe test on this one, not a benchmark, the real benchmarks are listed. My test shows what the models can do when both tasked with a long-running, technically difficult, one-shot task. I think your test you describe (collaborative, task delegation, task completion, TTD, steerability) is a great format for a future test that I will definitely try out.

The important point is that your benchmark is pretty much irrelevant for the actual usage. Thus whatever conclusion you draw is not just irrelevant but misleading.

Re: GLM 5.2 vs. Opus

#282

Post by 59nadir »

jameswhitford wrote: post #83
close2 wrote: post #39

I wonder how much tokens and time where used for the verifying part. Maybe GLM 5.2 instantly found the "solution" to read the screen pixel by pixel, but it could also have been a major token and time consumer.

Hi, author here, I cannot give an exact number for how many token the verification step took, but the verification GLM 5.2 ran was very stupid and definitely a waste of time. It read the pixel color data to try and verify the scene rendered properly. Which is really bad. Opus opened the game in a Playwright browser and took screenshots to verify the actual image. Which helped a lot. Pro tip: You could use a multi-mod…

That's a dumb way to do it, it should just write the frame buffer to a PNG instead of taking screenshots. I guess you can't take the dumb web developer ways out of these models at the end of the day.

Re: GLM 5.2 vs. Opus

#283

Post by canes123456 »

cultofmetatron wrote: post #19

I seriously dont' know all this big hullabaloo about one shot prompting. by definition, a single prompt wont' constitute the complexity of a software project. ergo, what you'll get is a series of assumptions made by the model based on preexisting code in its training corpus. I'd rather see a coding agent that can follow steps in a plan file to a T while following guardrails and adhering to the proper coding conventio…

Isn't a plan file just a single long prompt?

Re: GLM 5.2 vs. Opus

#284

Post by hintymad »

cultofmetatron wrote: post #19

I seriously dont' know all this big hullabaloo about one shot prompting. by definition, a single prompt wont' constitute the complexity of a software project. ergo, what you'll get is a series of assumptions made by the model based on preexisting code in its training corpus. I'd rather see a coding agent that can follow steps in a plan file to a T while following guardrails and adhering to the proper coding conventio…

> I'd rather see a coding agent that can follow steps in a plan file to a T while following guardrails and adhering to the proper coding conventions in the human reviewed spec.

In fact, I'd rather see Anthropic publish a convincing project that does this using Claude. The project should be complex enough and novel enough to show the world how reliable and powerful Claude is. That is, Anthropic does not need Amodei or its employees to tell us that whatever percent of engineers will lose their jobs. They can just show us. Easily.

Re: GLM 5.2 vs. Opus

#285

Post by nfriedly »

ijidak wrote: post #248
esperent wrote: post #179

Earlier quoted context omitted.

My pi usage over the past ~5 months went roughly like this: * Install pi and a bunch of extensions from their package repo * Realize that all the packages (with a few exceptions) are massively overcomplicated and vibe coded * Ask pi to rebuild a very simple version of the packages I used. So e.g. subagents - all the default subagent extensions are massively complicated with named agents, recursion, communication. I m…

What type of task are you running for ten hours? Is this a programming task? I've not come across a programming task that would take an LLM ten hours.

I'm not the person you asked, but if they're running in their own local hardware, then it might just be a lot slower than what the big providers run their models on. System RAM is a lot cheaper than VRAM, especially if you bought it last year.

Re: GLM 5.2 vs. Opus

#286

Post by cmrdporcupine »

faxmeyourcode wrote: post #274

I feel like another comparison worth looking at is purely cost. Capability per dollar is something I care about: Opus API $5/$25 Sonnet API $5/$15 Haiku API $1/$5 GLM 5.2 API $1.4/$4.4 So you're really getting near opus level capability for the price of haiku .

Not really, GLM uses more tokens to get work done.

Re: GLM 5.2 vs. Opus

#287

Post by johnnyApplePRNG »

InsideOutSanta wrote: post #272

One nice thing about GLM is that it has never refused a task. I'm working on a website that renders countries right now, and Anthropic's models regularly give me the old "This request triggered safety guardrails." I'm not sure what exactly triggers it, but it seems to happen when it has to look at lists of countries. I suspect there must be at least one country name that triggers the safety guardrail. You'd expect GL…

The amount of times I have had to spend tokens to attempt (in futility) to convince a proprietary model that the request I asked it to perform on code that I wrote is safe/legal/moral is insane.

Part of me wants to believe they really do care about protecting the world from... something... I don't know quite what exactly tbh... but it must be costing them a small fortune to scan each input and output against N guardrails and they are a for-profit corporation who could easily turn a blind eye to all of this and simply say "what you do with this model is on you" like I would expect most corporations to.

Strange times.

Re: GLM 5.2 vs. Opus

#288

Post by jameson »

> Opus 4.8 built in Claude Code; GLM-5.2 built in Pi over OpenRouter.

It would be more interesting and accurate to see the comparison on the same harness if the intent is to compare the frontier models.

Pi is relatively new and does not have many features built-in compared to Claude Code. It was chosen intentionally this way as Pi's goal is not to create a bloat builtin of tools most don't use but to allow the users to customize to fit their need -- similar to Neovim vs IDE.

The end-user "vibe coding" experience is *heavily* swayed by the harness because prompt effectively drives how a model outputs an answer.

Re: GLM 5.2 vs. Opus

#289

Post by yard2010 »

david_shi wrote: post #24

> GLM-5.2 cost a fraction as much. Opus finished in half the time and shipped a cleaner game. Off topic, but does anyone else instantly pick up on LLMisms like this? It seems like all the models have converged on this style of writing, and improvements aren't really changing it.

I cannot unsee it.

There was this dude here not long ago who bought like $70k worth of gpus to research, and if I'm not mistaken his research was something related to make llms sound less llm-y. I wonder how it goes for him.

Re: GLM 5.2 vs. Opus

#290

Post by wiremine »

cmrdporcupine wrote: post #286
faxmeyourcode wrote: post #274

I feel like another comparison worth looking at is purely cost. Capability per dollar is something I care about: Opus API $5/$25 Sonnet API $5/$15 Haiku API $1/$5 GLM 5.2 API $1.4/$4.4 So you're really getting near opus level capability for the price of haiku .

Not really, GLM uses more tokens to get work done.

I ran a fairly large experiment last week, and the token usage wasn't bad at all. What softs of use cases are you seeing large token usage by GLM 5.2?
Post reply on HN