The upcoming GPT-3 moment for RL • Hacker News Board

Re: The upcoming GPT-3 moment for RL

#41

Post by phillipcarter » Sun, Jul 13, 2025, 4:45 PM UTC

ademup wrote: post #26

phillipcarter wrote: post #18

Earlier quoted context omitted.

> The main barrier is cost I very much disagree. For the larger, more sophisticated stuff that runs our world, it is not cost that prohibits wide and deep automation. It's deeply sophisticated and constrained requirements, highly complex existing behaviors that may or may not be able to change, systems of people who don't always hold the information needed, usually wildly out of date internal docs that describe the s…

The status quo system you describe isn't objectively optimal. It sounds archaic to me. "We" would never intentionally design it this way if we had a fresh start. I believe it is this way due to a meriad of reasons, mostly stemming from the frailty and avarice of people. I'd argue the opposite of your stance: we've never had a chance at a fresh start without destruction, but agents (or their near-future offspring) can…

Agents quite literally cannot do this today.

Additionally, I disagree with your point:

> The status quo system you describe isn't objectively optimal.

On the basis that I would challenge you or anyone to judge what is objectively optimal. Google Search is a wildly complex system, an iceberg or rules on top of rules specifically because it is a digital infrastructure surrounding an organic system filled with a diverse group of people with ever-changing preferences and behaviors. What, exactly, would be optimal here?

Re: The upcoming GPT-3 moment for RL

#42

Post by brookst » Sun, Jul 13, 2025, 4:46 PM UTC

kubb wrote: post #40

brookst wrote: post #32

Earlier quoted context omitted.

There are also two kinds of people - those who are objective enough to tell when it happens and those who will never even see when they’re outperformed because of their cognitive biases. I give you a 2x2x2 matrix.

Sure, but if a person can find an easier way to do their job, they’ll usually do it. Usually the bias is towards less energy expenditure.

For many people, yes. For people who have their identity invested in being the smartest person in the room, life is considerably harder.

Re: The upcoming GPT-3 moment for RL

#43

Post by m3kw9 » Sun, Jul 13, 2025, 4:47 PM UTC

Upcoming sure, everything can be upcoming like the famous ASI

Re: The upcoming GPT-3 moment for RL

#44

Post by levzzz » Sun, Jul 13, 2025, 4:51 PM UTC

esperent wrote: post #36

Szpadel wrote: post #7

with RL it's hard to define score function in many categories. rhis is especially visible in current coding capabilities. LLM will very often create sloppy solutions because they work well in RL. hardcoding API keys? ignoring errors? disabling lints? those pass in automated evaluation therefore are reinforced in training. are they good solutions? of course not. It's very hard to define (in way to create lints) what m…

> hardcoding API keys? ignoring errors? disabling lints? These kind of "rookie mistakes" are not things that any modern LLM is likely to do. Indeed, I had to argue quite strongly with Gemini recently when I was learning a new tool (so basically just playing around with a fully local setup) and I hardcoded an API key then tried to commit it. The LLM did NOT like that! I had to carefully explain that this was a toy rep…

[dead]

Re: The upcoming GPT-3 moment for RL

#45

Post by TheOtherHobbes » Sun, Jul 13, 2025, 4:51 PM UTC

brookst wrote: post #33

amluto wrote: post #22

Earlier quoted context omitted.

The best part when a “thinking” model carefully thinks and then says something that is obviously illogical, when the model clearly has both the knowledge and context to know it’s wrong. And then you ask it to double check and you give it a tiny hint about how it’s wrong, and it profusely apologizes, compliments you on your wisdom, and then says something else dumb. I fully believe that LLMs encode enormous amounts of…

They key difference between that and humans, if course, is that most humans will double down on their error and insist that your correction is wrong, throwing a kitchen sink of appeals to authority, motte/bailey, and other rhetorical techniques at you.

That's not any different in practice to the LLM "apologising" to placate you and then making a similar mistake again.

It's not even a different strategy. It's just using rhetoric in a more limited way, and without human emotion.

These are style over substance machines. Their cognitive abilities are extremely ragged and unreliable - sometimes brilliant, sometimes useless, sometimes wrong.

But we give them the benefit of the doubt because they hide behind grammatically correct sentences that appear to make sense, and we're primed to assume that language = sentience = intelligence.

Re: The upcoming GPT-3 moment for RL

#46

Post by cjblomqvist » Sun, Jul 13, 2025, 5:02 PM UTC

kevindamm wrote: post #23

gcanyon wrote: post #21

The "GPT-3 moment" framing is a bit hype-y I think? GPT-3 eliminated the need for task-specific fine-tuning, but from the article RL wouldn't replace LLM-style pretraining. So this is more of an incremental advance than the paradigm shift GPT-3 represented. That said, if it unlocks RL generalization that would be huge. The core claim that massive-scale RL will unlock generalization doesn't seem that surprising since…

It's a very big "if" because other fields are comparatively underspecified. There's no equivalent to a compiler or interpreter in most cases (with spreadsheets being the lingua franca that comes even close for most industries). It would "work" but I think it will need even more scrutiny by experts to confirm what's correct and what needs to be re-generated. Please please no vibe accounting.

> Please please no vibe accounting.

Funny you mention; There are multiple companies in Sweden working on AI/ML based accounting. It's not so different from AI/ML based automated driving.

Re: The upcoming GPT-3 moment for RL

#47

Post by majormajor » Sun, Jul 13, 2025, 5:09 PM UTC

ademup wrote: post #26

phillipcarter wrote: post #18

Earlier quoted context omitted.

> The main barrier is cost I very much disagree. For the larger, more sophisticated stuff that runs our world, it is not cost that prohibits wide and deep automation. It's deeply sophisticated and constrained requirements, highly complex existing behaviors that may or may not be able to change, systems of people who don't always hold the information needed, usually wildly out of date internal docs that describe the s…

The status quo system you describe isn't objectively optimal. It sounds archaic to me. "We" would never intentionally design it this way if we had a fresh start. I believe it is this way due to a meriad of reasons, mostly stemming from the frailty and avarice of people. I'd argue the opposite of your stance: we've never had a chance at a fresh start without destruction, but agents (or their near-future offspring) can…

It's not claimed to be an "objectively optimal" solution, it's claimed to represent how the world works.

I don't know where you're going with discussion of destruction and killing, but even fairly simple consumer products have any number of edge cases that initial specifications rarely capture. I'm not sure what "objectively optimal" is supposed to mean here, either.

If a spec described every edge case it would basically be executable already.

The pain of developing software at scale is that you're creating the blueprint on the fly from high-level vague directions.

Something trivial that nevertheless often results in meetings and debate in the development world:

Spec requirement 1: "Give new users a 10% discount, but only if they haven't purchased in the last year."

Spec requirement 2, a year later: "Now offer a second product the user can purchase."

Does the 10% discount apply to the second product too? Do you get the 10% discount on the second product if you purchased the first product in the last year, or does a purchase on any product consume the discount eligibility? What if the prices are very different and customers would be pissed off if a $1 discount on the cheaper product (which didn't meet their needs in the end) prevented them from getting a 10$ discount 9 months later (which they think will)? What if the second product is a superset of the first product? What if there are different relevant laws in different jurisdictions where you're selling your product?

Agents aren't going to figure out the intent of the company's principal's automatically here because the decision maker doesn't actually even realize it's a question until the implementers get into the weeds.

A sufficiently advanced agent would present all the options to the person running the task, and then the humans could decide. But then you've slowed things back down the pace of the human decision makers.

The complexities only increase as the product grows. And once you get into distributed or concurrent systems even most of our code today is ambiguous enough about intent that bugs are common.

Re: The upcoming GPT-3 moment for RL

#48

Post by curious_cat_163 » Sun, Jul 13, 2025, 5:35 PM UTC

> Rather than fine-tuning models on a small number of environments, we expect the field will shift toward massive-scale training across thousands of diverse environments.

This is a great hypothesis for you to prove one way or the other.

> Doing this effectively will produce RL models with strong few-shot, task-agnostic abilities capable of quickly adapting to entirely new tasks.

I am not sure if I buy that, frankly. Even if you were to develop radically efficient means to create "effective and comprehensive" test suites that power replication training, it is not at all a given that it will translate to entirely new tasks. Yes, there is the bitter lesson and all that but we don't know if this is _the_ right hill to climb. Again, at best, this is a hypothesis.

> But achieving this will require training environments at a scale and diversity that dwarf anything currently available.

Yes. You should try it. Let us know if it works. All the best!

Re: The upcoming GPT-3 moment for RL

#49

Post by copypaper » Sun, Jul 13, 2025, 6:29 PM UTC

kuruczgy wrote: post #16

mohsen1 wrote: post #5

I’ve been exploring this too, since I rely on LLMs a lot to build software. I’ve noticed that our dev loop-writing, testing-is often mostly human-guided, but language models frequently outperform us in reasoning. If we plug in more automation; MCP tools controlling browsers, documentation readers, requirement analysers, we can make the cycle much more automated, with less human involvement. This article suggests scal…

> but language models frequently outperform us in reasoning what 99% of the time their reasoning is laughable. Or even if their reasoning is on the right track, they often just ignore it in the final answer, and do the stupid thing anyway.

Yea I don't understand how people are "leaving it running overnight" to successfully implement features. There just seems to be a large disconnect between people who are all in on AI development and those who aren't. I have a suspicion that the former are using Python/JS and the features they are implementing are simple CRUD APIs while the latter are using more than simple systems/languages.

I think the problem is that despite feeding it all the context and having all the right MCPs agents hooked up, is that there isn't a human-in-loop. So it will just reason against itself causing these laughable stupid decisions. For simple boilerplate tasks this isn't a problem. But as soon as the scope is outside of a CRUD/boilerplate problem, the whole thing crumbles.

Re: The upcoming GPT-3 moment for RL

#50

Post by visarga » Sun, Jul 13, 2025, 6:36 PM UTC

This works great for software, math and games where you can have cheap validation. But what about messy real world tasks? I think hindsight learning from chat logs could fit the bill. What do I mean?

Imagine a long conversation. It is hard to judge if an AI response was useful or not immediately, but if you know the following 20 messages, it might be easy to infer. Not only you can see how it went, but sometimes you get real world validation.

For example a user comes to a LLM with a task, takes an idea, tries it in reality. Later they return, maybe in a new chat session, and continue iterating. You get real world testing of LLM responses through people.

This can be used to generate "preference scores", and train a preference model, with which you can do RLHF. So the user privacy is protected.

I call this the human-AI experience flywheel. Of course the larger the user base, the more experience the model collects. At the moment OpenAI has 500M users, they probably generate 0.5T interactive tokens/day. Those tokens go both into human brains and LLM logs.

It’s not about environment engineering anymore, it's about consequence harvesting. Meaningful validation emerges from systems actually being used by humans for real purposes.