Earlier quoted context omitted.
A rank-3 tensor, yes. Matrices are rank-2 tensors.
God I hated tensors in grad school. Give me a Taylor series any day.
Everything that seems complicated is just a "fancy matrix".
The upcoming GPT-3 moment for RL
81–90 of 104 posts
Re: The upcoming GPT-3 moment for RL
#82What is RL? Real Life? Or Reinforced Learning?
Real Life would be IRL.
Re: The upcoming GPT-3 moment for RL
#83Isn't "replication training" just adversarial training, as in GANs?
One network tries to clean-room implement the hash function. The other network tries to find an input for which the reference implementation behaves differently.
Re: The upcoming GPT-3 moment for RL
#84Upcoming sure, everything can be upcoming like the famous ASI
When I see all the bad verbal reasoning being spewed around LLMs it becomes easier to understand why so many people think these LLMs are intelligent.
Non rigorous reasoning is at the root of the problem here. AI hype is often indistinguishable fro AI slop, because those who believe it are also not very good at demanding and asserting rigor.
Re: The upcoming GPT-3 moment for RL
#85Think we will see something like this in the near future. But there are two very bold claims: one is hope that RL will lead to generalization in other domains, which based on current evidence seems far fetched. Other is the softwares and specs are good "RL-gym" data. The whole idea behind RL is that model explores the best paths, but if the softwares written are suboptimal when it comes to agent interaction paradigm (they were written for humans, not agents), there is a high chance even with the compute the model would be suboptimal. There is a parallel trend where current AI systems are abstracting entire workflows. Not accounting for that would lead to outcomes which are not cognizant of the current requirements.
Re: The upcoming GPT-3 moment for RL
#86They have a point about RLs increasing importance. From my outsider perspective, all major advances in model capabilities in the last period of time come from RL, so it's natural to expect that we can "milk" RL more for performance gains. Scaling RL is a natural way to attempt that. What I don't necessarily see is the generalization factor - say, we improve software engineering and math performance through RL learnin…
One detail the OP glosses over is the increasing costs of RL as the sequence length increases. If we’re just reasoning through an simple arithmetic problem, it’s a pretty manageable number of reasoning tokens and answer tokens. For a complete piece of software the answer might be 10 million tokens, and that doesn’t even count the reasoning. Now imagine that there was a mistake at some point. The model will need to go…
I'd expect that's manageable by some sort of agent-of-agent pattern. You have a high-level planning instance that calls upon fresh LLM instances (new context window!) for executing more targeted tasks or bug-fixes.
Currently, an LLM with everything under the sun in the context window behaves rather poorly and gets confused by that, even if we're not exceeding the context window length. Although it'd be certainly also interesting to train for increasing the maximum _actually_ usable context window length, I don't know how feasible that would be.
Re: The upcoming GPT-3 moment for RL
#87I’ve been exploring this too, since I rely on LLMs a lot to build software. I’ve noticed that our dev loop-writing, testing-is often mostly human-guided, but language models frequently outperform us in reasoning. If we plug in more automation; MCP tools controlling browsers, documentation readers, requirement analysers, we can make the cycle much more automated, with less human involvement. This article suggests scal…
> but language models frequently outperform us in reasoning what 99% of the time their reasoning is laughable. Or even if their reasoning is on the right track, they often just ignore it in the final answer, and do the stupid thing anyway.
Yes, if a LLM outperforms you, you have never reasoned in your life.
I will assume you passed high-school based on your looks and not on your abilities.
Re: The upcoming GPT-3 moment for RL
#88The "GPT-3 moment" framing is a bit hype-y I think? GPT-3 eliminated the need for task-specific fine-tuning, but from the article RL wouldn't replace LLM-style pretraining. So this is more of an incremental advance than the paradigm shift GPT-3 represented. That said, if it unlocks RL generalization that would be huge. The core claim that massive-scale RL will unlock generalization doesn't seem that surprising since…
It's a very big "if" because other fields are comparatively underspecified. There's no equivalent to a compiler or interpreter in most cases (with spreadsheets being the lingua franca that comes even close for most industries). It would "work" but I think it will need even more scrutiny by experts to confirm what's correct and what needs to be re-generated. Please please no vibe accounting.
Accounting, specifically book-keeping, really plays to the strengths of LLMs - pattern matching within a bounded context.
The primary task in book-keeping is to classify transactions (from expense vouchers, bank transactions, sales and purchase invoices and so on) and slot them into the Chart of Accounts of the business.
LLMs can already do this well without any domain/business specific context. For example - a fuel entry is so obvious that they can match it into a similar sounding account in the CoA.
And for others where human discretion is required, we can add a line of instruction in the prompt, and that classification is permanently encoded. A large chunk of these kind of entries are repetitive in nature, and so each such custom instruction is a long-term automation.
You might have not been speaking about simple book-keeping. If so, I'm curious to learn.
Re: The upcoming GPT-3 moment for RL
#89The issue is that it is a stretch to call it reinforcement learning when all we do currently (in the context of LLMs) is to multiply the reward with the learning rate.
It sounds cool as marketing. It helps improve LLMs a bit. And it will never yield s.th. like an AGI or anything that is "reasoning". Unless you also redefine the word reasoning of course.
Re: The upcoming GPT-3 moment for RL
#90Excellent commentary from Kaparthy on RL: https://x.com/karpathy/status/1944435412489171119