Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

arxiv.org

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

#21

Post by ndr »

janalsncm wrote: post #15

I really feel out of my depth because 2 out of the 3 methods here seem like they shouldn’t work? > To evaluate comprehensibility quantitatively, we employ an LLM-as-a-Judge framework This isn’t the worst idea, but it’s still a bit incestuous. Adding an LLM judge to check for hallucinations creates two new kinds of problems: false positives, where your judge hallucinates an incorrect fact, and false negatives, where t…

You can and should eval your judges. They're also typically easier to eval because often you have them emit categorical/structured data.

Re: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

#22

Post by janalsncm »

hansvm wrote: post #17
janalsncm wrote: post #15

I really feel out of my depth because 2 out of the 3 methods here seem like they shouldn’t work? > To evaluate comprehensibility quantitatively, we employ an LLM-as-a-Judge framework This isn’t the worst idea, but it’s still a bit incestuous. Adding an LLM judge to check for hallucinations creates two new kinds of problems: false positives, where your judge hallucinates an incorrect fact, and false negatives, where t…

> LLM-as-a-Judge Empirically, many problems look like they're easier to check than they are to solve. This seems like a reasonable way to bootstrap a little extra performance, with prior art in well-known DeepMind experiments. It's unclear if it works recursively (I imagine not), but the core idea is solid.

I’m not reacting to the idea of using an LLM as a judge in general. That’s a proven path.

I’m specifically reacting to using it to reward the chain of thought during RL training because models love to hack their rewards, learning any possible shortcut rather than the task we want them to.

Re: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

#23

Post by empiricus »

plastic-enjoyer wrote: post #2

> scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; Man, I find SOTA deep learning somewhat hilarious. We scale models to absurd proportions, burning through a shitload of resources just to achieve (slightly above) human intelligence. The human brain has a few billion neurons and uses as much power as a light bulb.

Training a model is more like taking an alien (from a different universe) and teaching it a huge chunk of the entire human civilization. A better comparison would be how much energy it takes for a human to produce a page a text (roughly the equivalent of 1hour of thinking?), and how much energy it takes for an LLM. GPT Sol estimates a frontier LLM is 10-50x times more efficient than a human at generating a page of text.

Re: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

#24

Post by myrmidon »

pfdietz wrote: post #3
plastic-enjoyer wrote: post #2

> scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; Man, I find SOTA deep learning somewhat hilarious. We scale models to absurd proportions, burning through a shitload of resources just to achieve (slightly above) human intelligence. The human brain has a few billion neurons and uses as much power as a light bulb.

The human brain is estimated to have approaching 10^11 neurons (most of them in the cerebellum). However, a neuron is much more than a single parameter. The brain is estimated to have from 10^14 to 5x10^14 synapses.

I think its fair to argue that modern silicon easily makes up the difference with clock speed; neurons can't do Ghz things by design (signal propagation in biological brains is also really slow compared to copper).

I'm pretty confident that algorithm/hardware design insights over the next decades will allow us to build human-rivaling cognitive abilities on basically todays consumer tech (analogously to how computer chess improved over time).

Re: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

#25

Post by vidarh »

plastic-enjoyer wrote: post #18
mapontosevenths wrote: post #13

Earlier quoted context omitted.

This is the worst they will ever be. In 1956 a 5mb hard drive shipped on a large truck and took a team of men to unload. It consumed huge amounts of power, and cost about $3,200/month to run. In today's dollars that would be about $160,000 per month. Aren't you glad we didnt just give up because it was kind of expensive?

Nobody's saying give up. I'm saying if your solution needs a trillion parameters and a power plant, and biology does it with a few billion neurons and a sandwich, that you're maybe on the wrong track. This is not an engineering gap.

Typical storage densities have improved by several magnitudes per decade for decades. A mere 3 magnitudes of gap doesn't just seem like an engineering gap, but a relatively small one.

Re: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

#26

Post by K0balt »

plastic-enjoyer wrote: post #2

> scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; Man, I find SOTA deep learning somewhat hilarious. We scale models to absurd proportions, burning through a shitload of resources just to achieve (slightly above) human intelligence. The human brain has a few billion neurons and uses as much power as a light bulb.

GPUs are fundamentally a terribly inefficient way to achieve our goal, the calculations we are doing with them are difficult problems that fall out of physics for free. Tech like thermal wells could give us 2t models the size of micro-SD cards that consume milliwatts. There are already commercially available ASIC LLM models (the model is burned into silicon) that use 100x less power than running it on a GPU, while allowing a single card to serve thousands of users at once.

There is a lot of low hanging fruit here as models stabilize, and the more ambitious tech that might take a decade or so to land offers efficiencies 10-100x compared even to biological systems. (Most of it requires cryogenic temperatures though). A forward looking tech investment might be inexpensive, tiny, efficient cryogenic cooling systems optimized for desktop or portable use.

Re: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

#27

Post by pennomi »

vidarh wrote: post #25
plastic-enjoyer wrote: post #18

Earlier quoted context omitted.

Nobody's saying give up. I'm saying if your solution needs a trillion parameters and a power plant, and biology does it with a few billion neurons and a sandwich, that you're maybe on the wrong track. This is not an engineering gap.

Typical storage densities have improved by several magnitudes per decade for decades. A mere 3 magnitudes of gap doesn't just seem like an engineering gap, but a relatively small one.

I think the criticism is not “we need to have better hardware that will shrink over time”, it’s “our algorithms are hilariously inefficient, and nature shows that a better way must exist”.

Maybe there’s also a hardware component to it, but there’s very little point in trying to optimize the hardware to work with a poor algorithm. Once we discover an efficient way to train and infer, then it will be worth hyper-engineering the hardware.

Re: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

#28

Post by vidarh »

pennomi wrote: post #27
vidarh wrote: post #25

Earlier quoted context omitted.

Typical storage densities have improved by several magnitudes per decade for decades. A mere 3 magnitudes of gap doesn't just seem like an engineering gap, but a relatively small one.

I think the criticism is not “we need to have better hardware that will shrink over time”, it’s “our algorithms are hilariously inefficient, and nature shows that a better way must exist”. Maybe there’s also a hardware component to it, but there’s very little point in trying to optimize the hardware to work with a poor algorithm. Once we discover an efficient way to train and infer, then it will be worth hyper-engine…

The point of the hardware comparison was not to argue that the hardware should be optimised, but to point out that it's ludicrous to point to the scale of the difference as evidence it's not an engineering gap when we have a history of overcoming far greater gaps in scale.

Yes, we need better architectures and algorithms. We can point to massive advances in software as well in many spaces, including in LLMs (e.g. compare early GPT versions with current smaller open models), but the hardware comparison came from further up-thread.

Re: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

#29

Post by gertlabs »

janalsncm wrote: post #15

I really feel out of my depth because 2 out of the 3 methods here seem like they shouldn’t work? > To evaluate comprehensibility quantitatively, we employ an LLM-as-a-Judge framework This isn’t the worst idea, but it’s still a bit incestuous. Adding an LLM judge to check for hallucinations creates two new kinds of problems: false positives, where your judge hallucinates an incorrect fact, and false negatives, where t…

LLM as judge / self-distillation is effective insofar as it can make models more reliably do things they are already capable of. But I agree that for pushing the frontier of what a model is capable of understanding and producing, incestuous is a good word and it's unlikely to scale far.
Post reply on HN