Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The upcoming GPT-3 moment for RL

mechanize.work

Post reply on HN
Print view View original

11–20 of 104 posts

Re: The upcoming GPT-3 moment for RL

#11

Post by clbrmbr »

criemen wrote: post #4
amelius wrote: post #3

Step 1. Train a VLM to supervise the RL training. Step 2. Train the RL network. In the mean time drink coffee or work on plan of world domination.

My understanding is that this is essentially how RLHF works, and it doesn't scale. As you run RL for longer, the model will learn how to cheat the imperfections of the grader, instead of getting better at the task at hand. Therefore, to scale RL you really need good graders, and determinism is king.

Do you think constitutional approaches would help here? (Verifiable reward for the main score, but then asking the model to self-critique for security and quality.)

Re: The upcoming GPT-3 moment for RL

#12

Post by szundi »

YetAnotherNick wrote: post #9
mohsen1 wrote: post #5

I’ve been exploring this too, since I rely on LLMs a lot to build software. I’ve noticed that our dev loop-writing, testing-is often mostly human-guided, but language models frequently outperform us in reasoning. If we plug in more automation; MCP tools controlling browsers, documentation readers, requirement analysers, we can make the cycle much more automated, with less human involvement. This article suggests scal…

RL is a training method and it improves the model itself. So basically one step(e.g. successful test run, finding search result) could create positive and negative examples for the other step(e.g. coding agent, search agent). And using this the base itself will improve to satisfy other demands and if it reaches close to 100% accuracy(which I believe it could as models mostly fail due to dumb mistakes in tests), you d…

[dead]

Re: The upcoming GPT-3 moment for RL

#15

Post by amelius »

criemen wrote: post #4
amelius wrote: post #3

Step 1. Train a VLM to supervise the RL training. Step 2. Train the RL network. In the mean time drink coffee or work on plan of world domination.

My understanding is that this is essentially how RLHF works, and it doesn't scale. As you run RL for longer, the model will learn how to cheat the imperfections of the grader, instead of getting better at the task at hand. Therefore, to scale RL you really need good graders, and determinism is king.

You're talking about training an LLM. I'm talking about training robotic/motor skills and haptic feedback.

Re: The upcoming GPT-3 moment for RL

#16

Post by kuruczgy »

mohsen1 wrote: post #5

I’ve been exploring this too, since I rely on LLMs a lot to build software. I’ve noticed that our dev loop-writing, testing-is often mostly human-guided, but language models frequently outperform us in reasoning. If we plug in more automation; MCP tools controlling browsers, documentation readers, requirement analysers, we can make the cycle much more automated, with less human involvement. This article suggests scal…

> but language models frequently outperform us in reasoning

what

99% of the time their reasoning is laughable. Or even if their reasoning is on the right track, they often just ignore it in the final answer, and do the stupid thing anyway.

Re: The upcoming GPT-3 moment for RL

#17

Post by kubb »

kuruczgy wrote: post #16
mohsen1 wrote: post #5

I’ve been exploring this too, since I rely on LLMs a lot to build software. I’ve noticed that our dev loop-writing, testing-is often mostly human-guided, but language models frequently outperform us in reasoning. If we plug in more automation; MCP tools controlling browsers, documentation readers, requirement analysers, we can make the cycle much more automated, with less human involvement. This article suggests scal…

> but language models frequently outperform us in reasoning what 99% of the time their reasoning is laughable. Or even if their reasoning is on the right track, they often just ignore it in the final answer, and do the stupid thing anyway.

There are 2 kinds of people. Those who are outperformed on their most common tasks by LLMs and those who aren’t.

Re: The upcoming GPT-3 moment for RL

#18

Post by phillipcarter »

mohsen1 wrote: post #5

I’ve been exploring this too, since I rely on LLMs a lot to build software. I’ve noticed that our dev loop-writing, testing-is often mostly human-guided, but language models frequently outperform us in reasoning. If we plug in more automation; MCP tools controlling browsers, documentation readers, requirement analysers, we can make the cycle much more automated, with less human involvement. This article suggests scal…

> The main barrier is cost

I very much disagree. For the larger, more sophisticated stuff that runs our world, it is not cost that prohibits wide and deep automation. It's deeply sophisticated and constrained requirements, highly complex existing behaviors that may or may not be able to change, systems of people who don't always hold the information needed, usually wildly out of date internal docs that describe the system or even how to develop for it, and so on.

Agents are nowhere near capable of replacing this, and even if they were, they'd change it differently in ways that are often undesirable or illegal. I get that there's this fascination with "imagine if it were good enough to..." but it's not, and the systems AI must exist in are both vast and highly difficult to navigate.

Re: The upcoming GPT-3 moment for RL

#19

Post by szundi »

Szpadel wrote: post #7

with RL it's hard to define score function in many categories. rhis is especially visible in current coding capabilities. LLM will very often create sloppy solutions because they work well in RL. hardcoding API keys? ignoring errors? disabling lints? those pass in automated evaluation therefore are reinforced in training. are they good solutions? of course not. It's very hard to define (in way to create lints) what m…

[dead]

Re: The upcoming GPT-3 moment for RL

#20

Post by CuriouslyC »

Szpadel wrote: post #7

with RL it's hard to define score function in many categories. rhis is especially visible in current coding capabilities. LLM will very often create sloppy solutions because they work well in RL. hardcoding API keys? ignoring errors? disabling lints? those pass in automated evaluation therefore are reinforced in training. are they good solutions? of course not. It's very hard to define (in way to create lints) what m…

You can project them onto a linear space by gathering enough pairwise evaluations. PelicanElo.
Post reply on HN