Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-5: Targeting complex systems engineering and long-horizon agentic tasks

z.ai

Post reply on HN
Print view View original

501–510 of 540 posts

Re: GLM-5: Targeting complex systems engineering and long-horizon agentic tasks

#501

Post by pama »

yuvadam wrote: post #498
jfaganel99 wrote: post #459

[flagged]

Do you mind explaining what this RL infrastructure actually entails and why it's a hard problem?

Not OP. Personal opinion on why it is a somewhat hard problem. The main problem is using the available compute correctly and productively while doing two very separate types of tasks that were previously solved independently: generating responses with llm inference engines and modifying weights with a training code. A step of training updates the weights so the inference engines have to adjust theirs, but we talk about 750B parameters and multiple inference servers. Stale weights can be used instead, but only for a tiny bit and the data from them needs special corrections that also involve large compute/memory. Your inference engines better be deterministic (for given pseudoRNG; it clashes with parallelism) or you have a way to correct the probability streams. Ideally inference and training should have same everything at the bit level when they handle the same context, but we dont live in that world yet. And of course, GPUs break. For no great reason, other than the tiny scale of their features making them fragile. And because you scale, you need to handle failures gracefully and efficiently.

Re: GLM-5: Targeting complex systems engineering and long-horizon agentic tasks

#503

Post by nananana9 »

vardalab wrote: post #339
dust42 wrote: post #233

Earlier quoted context omitted.

They are all just token generators without any intelligence. There is so little difference nowadays that I think in a blind test nobody will be able to differentiate the models - whether open source or closed source. Today's meme was this question: "The car wash is only 50 meters from my house. I want to get my car washed, should I drive there or walk?" Here is Claude's answer just right now: "Walk! At only 50 meters…

This is really just a meme. People don't know how to use these tools. Here is the response from Gpt-5.2 using my default custom instructions in the mac desktop app. OBJECTIVE: Decide whether to drive or walk to a car wash ~50 meters from home, given typical constraints (car must be present for wash). APPROACH: Use common car-wash workflows + short-distance driving considerations (warm engine, time, parking/queue). No…

> given typical constraints (car must be present for wash).

"It's very good at thinking if you do the thinking for it"

Re: GLM-5: Targeting complex systems engineering and long-horizon agentic tasks

#504

Post by zozbot234 »

pama wrote: post #501
yuvadam wrote: post #498

Earlier quoted context omitted.

Do you mind explaining what this RL infrastructure actually entails and why it's a hard problem?

Not OP. Personal opinion on why it is a somewhat hard problem. The main problem is using the available compute correctly and productively while doing two very separate types of tasks that were previously solved independently: generating responses with llm inference engines and modifying weights with a training code. A step of training updates the weights so the inference engines have to adjust theirs, but we talk abo…

Surely you could just pre-generate rollouts with slightly stale weights and then cheaply verify the rollout when up-to-date weights stream in by treating the former solution as speculative decoding. Sounds quite trivial to me, perhaps I'm missing something.

Re: GLM-5: Targeting complex systems engineering and long-horizon agentic tasks

#505

Post by pama »

zozbot234 wrote: post #504
pama wrote: post #501

Earlier quoted context omitted.

Not OP. Personal opinion on why it is a somewhat hard problem. The main problem is using the available compute correctly and productively while doing two very separate types of tasks that were previously solved independently: generating responses with llm inference engines and modifying weights with a training code. A step of training updates the weights so the inference engines have to adjust theirs, but we talk abo…

Surely you could just pre-generate rollouts with slightly stale weights and then cheaply verify the rollout when up-to-date weights stream in by treating the former solution as speculative decoding. Sounds quite trivial to me, perhaps I'm missing something.

Cheap verifying of speculative decoding only works for a few tokens at a time. Long sequence generations (thousands to tens of thousands of tokens in typical rollouts for thinking models) are dominated by distribution drift on stale weights (because slightly wrong probabilities multiply over long streams), and the off policy RL training methods dont work well (high variance) for such high dimensional problems.

Re: GLM-5: Targeting complex systems engineering and long-horizon agentic tasks

#507

Post by satvikpendem »

khimaros wrote: post #506
satvikpendem wrote: post #462

Earlier quoted context omitted.

2.5 is out now too.

i meant m2.1, but you are probably talking about kimi, not minimax

No, MiniMax M2.5 is now available on agent.minimax.io. We await the weights still.

Re: GLM-5: Targeting complex systems engineering and long-horizon agentic tasks

#508

Post by spmurrayzzz »

djsjajah wrote: post #408
spmurrayzzz wrote: post #308

Earlier quoted context omitted.

No I'm saying there are quite a few more bottlenecks than that (I/O being a big one). Even in the more efficient training frameworks, there's per-op dispatch overhead in python itself. All the boxing/unboxing of python objects to C++ handles, dispatcher lookup + setup, all the autograd bookkeeping, etc. All of the bottlenecks in sum is why you'd never get to 100% MFUs (but I was conceding you probably don't need to i…

That’s kind of a moot point. Even if none of those overheads existed you would still be getting a a fractions of the mfu. Models are fundamental limited by memory bandwidth even with best case scenarios of sft or prefill. And what are you doing that I/O is a bottleneck?

> That’s kind of a moot point.

I don't believe it's moot, but I understand your point. The fact that models are memory bandwidth bound does not at all mean that other overhead is insignificant. Your practical delivered throughput is the minimum of compute ceiling, bandwidth ceiling, and all the unrelated speed limits you hit in the stack. Kernel launch latency, Python dispatch, framework bookkeeping, allocator churn, graph breaks, and sync points can all reduce effective speed. There are so many points in the training and inference loop where the model isn't even executing.

> And what are you doing that I/O is a bottleneck?

We do a fair amount of RLVR at my org. That's almost entirely waiting for servers/envs to do things, not the model doing prefill or decode (or even up/down weighting trajectories). The model is the cheap part in wall clock terms. The hard limits are in the verifier and environment pipeline. Spinning up sandboxes, running tests, reading and writing artifacts, and shuttling results through queues, these all create long idle gaps where the GPU is just waiting to do something.

Re: GLM-5: Targeting complex systems engineering and long-horizon agentic tasks

#509

Post by itay-maman »

The inherent problem with evaluating coding performance of models remains: most day-to-day coding tasks are open-ended/partially-spec'd, and as such there is huge uncertainty on how the "right" solution looks.

It's very hard to rank models' solutions on such problems, which is why they rarely appear in benchmarks (I'd be glad to stand corrected).

Even Opus 4.5 coding a C compiler from scratch - jaw-dropping as it is - doesn't tell the whole story. Most of my tasks are not that well spec'd.

Re: GLM-5: Targeting complex systems engineering and long-horizon agentic tasks

#510

Post by zozbot234 »

spmurrayzzz wrote: post #508
djsjajah wrote: post #408

Earlier quoted context omitted.

That’s kind of a moot point. Even if none of those overheads existed you would still be getting a a fractions of the mfu. Models are fundamental limited by memory bandwidth even with best case scenarios of sft or prefill. And what are you doing that I/O is a bottleneck?

> That’s kind of a moot point. I don't believe it's moot, but I understand your point. The fact that models are memory bandwidth bound does not at all mean that other overhead is insignificant. Your practical delivered throughput is the minimum of compute ceiling, bandwidth ceiling, and all the unrelated speed limits you hit in the stack. Kernel launch latency, Python dispatch, framework bookkeeping, allocator churn,…

> That's almost entirely waiting for servers/envs to do things

I'm not sure why, sandboxes/envs should be small and easy to scale horizontally to the point where your throughput is no longer limited by them, and the maximum latency involved should also be quite tiny (if adequately optimized). What am I missing?

Post reply on HN