The upcoming GPT-3 moment for RL
mechanize.work
The upcoming GPT-3 moment for RL
1–10 of 104 posts
Re: The upcoming GPT-3 moment for RL
#2They have a point about RLs increasing importance. From my outsider perspective, all major advances in model capabilities in the last period of time come from RL, so it's natural to expect that we can "milk" RL more for performance gains.
Scaling RL is a natural way to attempt that.
What I don't necessarily see is the generalization factor - say, we improve software engineering and math performance through RL learning (probably easier for software engineering than math due to available training corpus). If that generalization factor doesn't hold, due the economics still work out? An expert-level software model would be useful to our profession, sure, but would it be enough to recoup the training costs if it's not applicable to other industries?
Re: The upcoming GPT-3 moment for RL
#3Step 1. Train a VLM to supervise the RL training.
Step 2. Train the RL network. In the mean time drink coffee or work on plan of world domination.
Re: The upcoming GPT-3 moment for RL
#4Step 1. Train a VLM to supervise the RL training. Step 2. Train the RL network. In the mean time drink coffee or work on plan of world domination.
My understanding is that this is essentially how RLHF works, and it doesn't scale. As you run RL for longer, the model will learn how to cheat the imperfections of the grader, instead of getting better at the task at hand.
Therefore, to scale RL you really need good graders, and determinism is king.
Re: The upcoming GPT-3 moment for RL
#5I’ve been exploring this too, since I rely on LLMs a lot to build software. I’ve noticed that our dev loop-writing, testing-is often mostly human-guided, but language models frequently outperform us in reasoning. If we plug in more automation; MCP tools controlling browsers, documentation readers, requirement analysers, we can make the cycle much more automated, with less human involvement.
This article suggests scaling up RL by exposing models to thousands of environments
I think we can already achieve something similar by chaining multiple agents:
1. A “requirement” agent that uses browser tools to craft detailed specs from docs.
2. A coding agent that sets up environments (Docker, build tools) via browser or CLI.
3. A testing agent that validates code against specs, again through tooling.
4. A feedback loop where the tester guides the coder based on results.
Put together, this system becomes a fully autonomous development pipeline-especially for small projects. In practice, I’ve left my machine running overnight, and these agents propose new features, implement them, run tests, and push to repo once they pass. It works surprisingly well.
The main barrier is cost—spinning up many powerful models is expensive. But on a modest scale, this method is remarkably effective.
Re: The upcoming GPT-3 moment for RL
#6> Each replication task consists of a detailed specification and a reference implementation. The central idea is that AI models are trained to produce an implementation that precisely matches the reference behavior. This clear-cut approach significantly simplifies evaluation, as the grading criteria are objective and direct: either the generated implementation behaves identically to the reference, or it doesn’t.
OK, but then you have to produce the detailed specification, working backward from the reference implementation. This is extremely non-trivial and it significantly weakens the TFA's parallels to pre-training, in which you don't need really need inputs other than raw text corpora.
I'm not saying this eliminates the idea outright, but I do think it hobbles it badly.
Re: The upcoming GPT-3 moment for RL
#7with RL it's hard to define score function in many categories.
rhis is especially visible in current coding capabilities.
LLM will very often create sloppy solutions because they work well in RL.
hardcoding API keys? ignoring errors? disabling lints? those pass in automated evaluation therefore are reinforced in training.
are they good solutions? of course not.
It's very hard to define (in way to create lints) what makes core readable and maintainable. Using other LLM for this task could cause original model to game the system by abusing some weaknesses in the other model.
for other tasks, how do you even evaluate thinks like eg user experience/app design? how to properly evaluate pelican ridding bicycle?
Re: The upcoming GPT-3 moment for RL
#8> Each replication task consists of a detailed specification and a reference implementation. The central idea is that AI models are trained to produce an implementation that precisely matches the reference behavior. This clear-cut approach significantly simplifies evaluation, as the grading criteria are objective and direct: either the generated implementation behaves identically to the reference, or it doesn’t. OK,…
The detailed specification is the output for a particular input.
And you can use a fuzzer to augument that.
Re: The upcoming GPT-3 moment for RL
#9I’ve been exploring this too, since I rely on LLMs a lot to build software. I’ve noticed that our dev loop-writing, testing-is often mostly human-guided, but language models frequently outperform us in reasoning. If we plug in more automation; MCP tools controlling browsers, documentation readers, requirement analysers, we can make the cycle much more automated, with less human involvement. This article suggests scal…
RL is a training method and it improves the model itself. So basically one step(e.g. successful test run, finding search result) could create positive and negative examples for the other step(e.g. coding agent, search agent). And using this the base itself will improve to satisfy other demands and if it reaches close to 100% accuracy(which I believe it could as models mostly fail due to dumb mistakes in tests), you don't need the testing agent altogether.
Re: The upcoming GPT-3 moment for RL
#10> Each replication task consists of a detailed specification and a reference implementation. The central idea is that AI models are trained to produce an implementation that precisely matches the reference behavior. This clear-cut approach significantly simplifies evaluation, as the grading criteria are objective and direct: either the generated implementation behaves identically to the reference, or it doesn’t. OK,…
When prompted correctly, models could generate good specification in form of pretty exhaustive tests. While all tests have weaknesses and are not formal specification, they could get us 99% there.