Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The upcoming GPT-3 moment for RL

mechanize.work

Post reply on HN
Print view View original

61–70 of 104 posts

Re: The upcoming GPT-3 moment for RL

#61

Post by jxmorris12 »

I recently wrote a post about scaling RL that has some similar ideas:

> How to Scale RL to 10^26 FLOPs (blog.jxmo.io/p/how-to-scale-rl-to-1026-flops)

The basic premise behind both essays is that for AI to make another big jump in capabilities, we need to find new data to train on.

My proposal was reusing text from the Internet and doing RL on next-token prediction. The linked post here instead suggests doing 'replication training', which they define as "tasking AIs with duplicating existing software products, or specific features within them".

Re: The upcoming GPT-3 moment for RL

#62

Post by rcbdev »

Zenst wrote: post #57

A few things that this made me think about: 1) If only there was a cryptocurrency tied to training AI models and make crypto grinding useful than maths that solve no real-world problem external to the token creation itself. 2) Larger and larger AI models, you start to get more hallucinations, maybe we should focus on dedicated highly tuned models for dedicated aspects and have a higher up conductor model that knows w…

I remember mining something called Gridcoin over a decade ago. It's a cryptocurrency tied to the BOINC project and rewards providing computing power to science.

I have sadly lost access to my wallet since.

Re: The upcoming GPT-3 moment for RL

#63

Post by magicalhippo »

brookst wrote: post #32
avs733 wrote: post #31

Earlier quoted context omitted.

there are also two kinds of people - those who are excited by that and those who are not. The result is a 2x2 matrix where several quadrants are deeply concerning to me.

There are also two kinds of people - those who are objective enough to tell when it happens and those who will never even see when they’re outperformed because of their cognitive biases. I give you a 2x2x2 matrix.

> I give you a 2x2x2 matrix.

That'd be a tensor, no?

Re: The upcoming GPT-3 moment for RL

#64

Post by ltbarcly3 »

This makes no sense. RL training data is predicated on past behavior of the agent. Whoever wrote this doesn't seem to fundamentally grasp what they are saying.

LLMs can be trained on an unsupervised way on static documents. That is really the key feature that lets them be as smart and effective as they are. If you had every other technology that LLMs are built on, and you didn't have hundreds of terabytes of text laying around, there would be no practical way to make them even a tiny tiny fraction as effective as they are currently.

Re: The upcoming GPT-3 moment for RL

#66

Post by sva_ »

magicalhippo wrote: post #63
brookst wrote: post #32

Earlier quoted context omitted.

There are also two kinds of people - those who are objective enough to tell when it happens and those who will never even see when they’re outperformed because of their cognitive biases. I give you a 2x2x2 matrix.

> I give you a 2x2x2 matrix. That'd be a tensor, no?

A rank-3 tensor, yes. Matrices are rank-2 tensors.

Re: The upcoming GPT-3 moment for RL

#68

Post by physix »

copypaper wrote: post #49
kuruczgy wrote: post #16

Earlier quoted context omitted.

> but language models frequently outperform us in reasoning what 99% of the time their reasoning is laughable. Or even if their reasoning is on the right track, they often just ignore it in the final answer, and do the stupid thing anyway.

Yea I don't understand how people are "leaving it running overnight" to successfully implement features. There just seems to be a large disconnect between people who are all in on AI development and those who aren't. I have a suspicion that the former are using Python/JS and the features they are implementing are simple CRUD APIs while the latter are using more than simple systems/languages. I think the problem is th…

I'd really like to know which use cases work and which don't. And when folks say they use agentic AI to churn through tokens to automate virtually the entire SDLC, are they just cherry picking the situations that turned out well, or do they really have prompting and workflow approaches that indeed increase their productivity 10-fold? Or, as you mention, is it possibly a niche area which works well?

My personal experience the past five months has been very mixed. If I "let 'er rip" it's mostly junk I need to refactor or redo by micro-managing the AI. At the moment, at least for what I do, AI is like a fantastic calculator that speeds up your work, but where you still should be pushing the buttons.

Re: The upcoming GPT-3 moment for RL

#69

Post by sva_ »

ltbarcly3 wrote: post #64

This makes no sense. RL training data is predicated on past behavior of the agent. Whoever wrote this doesn't seem to fundamentally grasp what they are saying. LLMs can be trained on an unsupervised way on static documents. That is really the key feature that lets them be as smart and effective as they are. If you had every other technology that LLMs are built on, and you didn't have hundreds of terabytes of text lay…

> Whoever wrote this doesn't seem to fundamentally grasp what they are saying.

RL != only online learning.

There's a ton of research on offline and imitation-based RL where the training data isn't tied to an agents past policy - which is exactly what this article is pointing to.

Re: The upcoming GPT-3 moment for RL

#70

Post by physix »

sva_ wrote: post #69
ltbarcly3 wrote: post #64

This makes no sense. RL training data is predicated on past behavior of the agent. Whoever wrote this doesn't seem to fundamentally grasp what they are saying. LLMs can be trained on an unsupervised way on static documents. That is really the key feature that lets them be as smart and effective as they are. If you had every other technology that LLMs are built on, and you didn't have hundreds of terabytes of text lay…

> Whoever wrote this doesn't seem to fundamentally grasp what they are saying. RL != only online learning. There's a ton of research on offline and imitation-based RL where the training data isn't tied to an agents past policy - which is exactly what this article is pointing to.

I'm not sufficiently familiar with the details on ML to assess the proposition made in the article.

From my understanding, RL is a tuning approach on LLMs, so the outcome is still the same kind of beast, albeit with a different parameter set.

So empirically, I actually thought that the lead companies would already be strongly focused on improving coding capabilities, since this is where LLMs are very effective, and where they have huge cashflows from token consumptions.

So, either the motivation isn't there, or they're already doing something like that, or they know it's not as effective as the approaches they already have.

I wonder which one it is.

Post reply on HN