I had the opportunity to meet Tamay not too long ago, very sharp guy. A lot of people I know are working on approaches to meta RL or exploration-based RL, where the goal is to build a foundation model of sorts with a really good world model across diverse tasks, and can predict good policies (or good policy updates) from limited rollouts and/or a sparse reward signal. We're not there quite yet, but as Altman recently said, "we don't have AGI until we have something that learns continuously", and there's a huge race in this space to make that happen.
The upcoming GPT-3 moment for RL
51–60 of 104 posts
Re: The upcoming GPT-3 moment for RL
#52Earlier quoted context omitted.
> but language models frequently outperform us in reasoning what 99% of the time their reasoning is laughable. Or even if their reasoning is on the right track, they often just ignore it in the final answer, and do the stupid thing anyway.
The best part when a “thinking” model carefully thinks and then says something that is obviously illogical, when the model clearly has both the knowledge and context to know it’s wrong. And then you ask it to double check and you give it a tiny hint about how it’s wrong, and it profusely apologizes, compliments you on your wisdom, and then says something else dumb. I fully believe that LLMs encode enormous amounts of…
I feel like one coding benchmark should be just telling it to double check or fix something that's actually perfectly fine repeatedly and watch how bad it deep fries your code base.
Re: The upcoming GPT-3 moment for RL
#53> Simple command-line tools that implement obscure hashing and encryption algorithms
So your plan is to train a MLP to black box replicate complex and highly non-linear encryption algorithms through gradient descent?
Re: The upcoming GPT-3 moment for RL
#54The "GPT-3 moment" framing is a bit hype-y I think? GPT-3 eliminated the need for task-specific fine-tuning, but from the article RL wouldn't replace LLM-style pretraining. So this is more of an incremental advance than the paradigm shift GPT-3 represented. That said, if it unlocks RL generalization that would be huge. The core claim that massive-scale RL will unlock generalization doesn't seem that surprising since…
This article stands as complete hype. They just seem to offer an idea of "replication training" which is just some vague agentic distributed RL. Multi-agent distributed reinforcement learning algorithms have been in the actual literature for a while. I suggest studying what DeepMind is doing for current state of the art in agentic distributed RL.
I didn’t think it was vague. Given an existing piece of software, write a detailed spec on what it does and then reward the model for matching its performance.
The vague part is whether this will generalize to other non software domains.
Re: The upcoming GPT-3 moment for RL
#55> Simple command-line tools that implement obscure hashing and encryption algorithms So your plan is to train a MLP to black box replicate complex and highly non-linear encryption algorithms through gradient descent?
Not black box, no. The spec presumably tells the model everything it needs to know or look up. But in contrast with Fibonacci, the exact code is unlikely to be in the training set verbatim.
Re: The upcoming GPT-3 moment for RL
#56They have a point about RLs increasing importance. From my outsider perspective, all major advances in model capabilities in the last period of time come from RL, so it's natural to expect that we can "milk" RL more for performance gains. Scaling RL is a natural way to attempt that. What I don't necessarily see is the generalization factor - say, we improve software engineering and math performance through RL learnin…
One detail the OP glosses over is the increasing costs of RL as the sequence length increases. If we’re just reasoning through an simple arithmetic problem, it’s a pretty manageable number of reasoning tokens and answer tokens.
For a complete piece of software the answer might be 10 million tokens, and that doesn’t even count the reasoning.
Now imagine that there was a mistake at some point. The model will need to go back to fix it, and understand the cascade of things the bugfix changed. It might be possible to keep that all in the context window but that seems like it won’t scale.
Re: The upcoming GPT-3 moment for RL
#57A few things that this made me think about:
1) If only there was a cryptocurrency tied to training AI models and make crypto grinding useful than maths that solve no real-world problem external to the token creation itself.
2) Larger and larger AI models, you start to get more hallucinations, maybe we should focus on dedicated highly tuned models for dedicated aspects and have a higher up conductor model that knows what to farm out to which models and from there combine and send out further requests to other models etc to come to a result. Certainly, the need for highly tuned niche models, after all, language recognition as an example, a model that could identify the language, local dialect and accent, that would then use a language model tuned better for that speaker it is recognising. That approach feels like the way over one large model that does it all itself.
Re: The upcoming GPT-3 moment for RL
#58Earlier quoted context omitted.
It's a very big "if" because other fields are comparatively underspecified. There's no equivalent to a compiler or interpreter in most cases (with spreadsheets being the lingua franca that comes even close for most industries). It would "work" but I think it will need even more scrutiny by experts to confirm what's correct and what needs to be re-generated. Please please no vibe accounting.
> Please please no vibe accounting. Funny you mention; There are multiple companies in Sweden working on AI/ML based accounting. It's not so different from AI/ML based automated driving.
I've seen some of those but all of the ones I've looked at also had a panel of experts who could give it a once-over (or re-work) before sending it back to the client. I'd compare it more to cruise control or driver-assist but not quite automated driving.
Re: The upcoming GPT-3 moment for RL
#59I like to imagine a world where we embraced nuclear and renewables where energy is free and we could throw gigawatthours at whatever crazy computation we can think of.
Re: The upcoming GPT-3 moment for RL
#60A few things that this made me think about: 1) If only there was a cryptocurrency tied to training AI models and make crypto grinding useful than maths that solve no real-world problem external to the token creation itself. 2) Larger and larger AI models, you start to get more hallucinations, maybe we should focus on dedicated highly tuned models for dedicated aspects and have a higher up conductor model that knows w…
For 1) isn't prime intellect doing that or something like it?