Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rich Sutton on AI creativity and discovery

twitter.com

Post reply on HN
Print view View original

131–140 of 141 posts

Re: Rich Sutton on AI creativity and discovery

#131

Post by arowthway »

simianwords wrote: post #119
arowthway wrote: post #65

Earlier quoted context omitted.

I don't think there is any "humans are metaphysically superior to LLMs" subtext to this talk, it's just a technical/educational observation. Access to some forms of evaluation and selective retention is inherent to humans and it's not inherent to LLMS. But it can be somehow bolted on and that's when they work best. It makes sense that more focus on those principles can yield better AI. I think the retention part is t…

> Access to some forms of evaluation and selective retention is inherent to humans and it's not inherent to LLMS I'm not sure I understood - what forms of evaluation is inherent to humans? If you don't give humans tools or access to the physical world, how can they evaluate?

There's no such thing as a human without access to the physical world.

Re: Rich Sutton on AI creativity and discovery

#132

Post by simianwords »

arowthway wrote: post #131
simianwords wrote: post #119

Earlier quoted context omitted.

> Access to some forms of evaluation and selective retention is inherent to humans and it's not inherent to LLMS I'm not sure I understood - what forms of evaluation is inherent to humans? If you don't give humans tools or access to the physical world, how can they evaluate?

There's no such thing as a human without access to the physical world.

So technically the only reason AI can't do discovery is access to physical word. When you give AI and humans access, they both do discoveries - that is the clean summary of the author's position.

Its not too interesting.. we already know that giving AI access to compilers and tools make them better.

Re: Rich Sutton on AI creativity and discovery

#133

Post by CGMthrowaway »

musebox35 wrote: post #52

The most successful applications like coding are not the result of pure LLM/generative modeling. They come from closing the loop with an agentic harness. The generate-test-selectively refine loop is the core modality of scientific work. An LLM + RL with Verifiable Rewards + feedback from compiler/terminal runs mimics this process to a great extend. This is Fisher/Box feedback loop ( https://www-sop.inria.fr/members/I…

> The generate-test-selectively refine loop is the core modality of scientific work.

Can you expand on this? Because when I think of it, it calls to mind p-hacking and selective publishing more than anything.

Like, when you try prompts over and over and over until you get code that finally works. At which point you stop and pronounce the dubious claim, "AI is amazing, look what it does!"

Re: Rich Sutton on AI creativity and discovery

#134

Post by fragebogen »

dang wrote: post #113
premieroncall wrote: post #91

Hmm, so the guy who said more data and compute will outperform any adhoc heuristics has shared a three step adhoc heuristic?

Can you please make your substantive points thoughtfully and without snark? I'm sure there is one here, but it's hard to make out what it is, and in any case the poison does more harm than the information does good. I'd link to the HN guidelines here but I'm on my phone!

Assuming OP meant "the bitter lesson" https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...

Re: Rich Sutton on AI creativity and discovery

#135

Post by highfrequency »

porridgeraisin wrote: post #130
highfrequency wrote: post #128

Earlier quoted context omitted.

The model’s distribution will certainly change from the base model’s output distribution during reinforcement learning, shifting toward outputs that score well on an external evaluation. This is very different from mode-seeking. Am I missing something?

Mode-seeking is describing the way in which the distribution changes. RL is capable of picking out slightly lower probability trajectories and moving them toward the top of the distribution. However, exploration is fundamentally limited by the base policy itself. If a trajectory has near-zero probability under the original model, RLVR is unlikely to discover it because it must first be sampled before it can be reward…

Sure, but I'd say that moving desirable trajectories from very low probability to high probability is characteristic of genuine human learning and discovery. Technically, quantum gravity, a bestselling novel, or a yet undiscovered proof of the Riemann Hypothesis is "in my distribution", but when we are talking about a long chain of unlikely token completions (with multiplicative probabilities), whether that trajectory lives in the tail of the distribution vs. in the mode makes all the difference.

Would you agree that it is a matter of degrees rather than a qualitative distinction? There seems to be a broad misconception in Sutton and others that output quality cannot exceed that of the base internet distribution; my point is that RL allows you to easily produce an output distribution that is better than whatever data you trained on according to some evaluation criteria. There are no clear theoretical limits on how much better it can get, rather there are many people asserting guesses that there is an upper bound and it lives below "human creativity." I just haven't seen any solid theoretical argument, and the empirical evidence has so far shown continual improvement.

Also, I would be keen to look at any sources you have of pass@k not improving much during GRPO.

Re: Rich Sutton on AI creativity and discovery

#136

Post by porridgeraisin »

highfrequency wrote: post #135
porridgeraisin wrote: post #130

Earlier quoted context omitted.

Mode-seeking is describing the way in which the distribution changes. RL is capable of picking out slightly lower probability trajectories and moving them toward the top of the distribution. However, exploration is fundamentally limited by the base policy itself. If a trajectory has near-zero probability under the original model, RLVR is unlikely to discover it because it must first be sampled before it can be reward…

Sure, but I'd say that moving desirable trajectories from very low probability to high probability is characteristic of genuine human learning and discovery. Technically, quantum gravity, a bestselling novel, or a yet undiscovered proof of the Riemann Hypothesis is "in my distribution", but when we are talking about a long chain of unlikely token completions (with multiplicative probabilities), whether that trajector…

I said slightly lower, I meant it. It's virtually impossible to sample a trajectory that is really really low probability (say, by smoothening the distribution before sampling) without incurring crazy amounts of noise. And only when you sample it, can you reward it and do the update.

Again, no one is saying models can't improve beyond the internet i.e data distribution! They clearly can. The claim is that RL without real exploration cannot exceed the base models distribution, which by virtue of SGD _does_ generalize.

And also, it doesn't mean it's not useful. Improving sample efficiency and making something that happens 1 in 15 times happen 1 in 1.2 times is insanely useful and is what has enabled the kind of coding agents we have today.

Sutton, especially, I doubt has a misconception about this :)

> pass@k

Yeah, AFK now. But it's a well researched thing. You can look for more, but here's one off the top of my head: https://openreview.net/forum?id=4OsgYD7em5 The original deepseek paper also had the result, i.e the paper that first got famous for using grpo as a method that works for LLMs. A side result in one of these papers I forget which one, is that the base model converges in performance with the RLd one at high k.

Re: Rich Sutton on AI creativity and discovery

#137

Post by dang »

fragebogen wrote: post #134
dang wrote: post #113

Earlier quoted context omitted.

Can you please make your substantive points thoughtfully and without snark? I'm sure there is one here, but it's hard to make out what it is, and in any case the poison does more harm than the information does good. I'd link to the HN guidelines here but I'm on my phone!

Assuming OP meant "the bitter lesson" https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...

I assume so as well but there isn't enough information in the comment to make a substantive point about it.

Re: Rich Sutton on AI creativity and discovery

#138

Post by highfrequency »

porridgeraisin wrote: post #136
highfrequency wrote: post #135

Earlier quoted context omitted.

Sure, but I'd say that moving desirable trajectories from very low probability to high probability is characteristic of genuine human learning and discovery. Technically, quantum gravity, a bestselling novel, or a yet undiscovered proof of the Riemann Hypothesis is "in my distribution", but when we are talking about a long chain of unlikely token completions (with multiplicative probabilities), whether that trajector…

I said slightly lower, I meant it. It's virtually impossible to sample a trajectory that is really really low probability (say, by smoothening the distribution before sampling) without incurring crazy amounts of noise. And only when you sample it, can you reward it and do the update. Again, no one is saying models can't improve beyond the internet i.e data distribution! They clearly can. The claim is that RL without…

Thanks, I appreciate the discussion. The paper you sent is interesting. I agree it looks like for moderate values of K (on the order of 100-1000), RL models actually look a little worse at pass@k than their base models.

So perhaps the right framing of your/Sutton's claim is: RL can upweight low-probability (p) but correct outputs, but there is a limit to how small p can be, and it is on the order of 1 in a 100 or 1 in a 1000. Implicitly there must be some crossover point where you would call this discovery/creativity if it works for sufficiently small p right? Eg if RL can upweight a correct but 1 in a trillion output to 1 in 5, that's got to count as discovery given that all possible sequences are technically "in the distribution"?

In practice, it does seem like that kind of progress is happening. For example with the recent Erdos solution [0], I would wager that GPT 4's hit rate on this would have been functionally 0 (certainly less than 1 in a thousand). Curious to hear whether you'd still say this is mode-seeking within a base distribution, or if not then what is the right explanation if not iterative RL.

I'd also highlight that the paper you linked with the pass@k equivalence doesn't technically address the question of how small p can be before RL upweighting breaks down - all of the example problems were easy enough that the base model had decent hit rate with 128 tries.

[0] https://openai.com/index/model-disproves-discrete-geometry-c...

Re: Rich Sutton on AI creativity and discovery

#139

Post by porridgeraisin »

highfrequency wrote: post #138
porridgeraisin wrote: post #136

Earlier quoted context omitted.

I said slightly lower, I meant it. It's virtually impossible to sample a trajectory that is really really low probability (say, by smoothening the distribution before sampling) without incurring crazy amounts of noise. And only when you sample it, can you reward it and do the update. Again, no one is saying models can't improve beyond the internet i.e data distribution! They clearly can. The claim is that RL without…

Thanks, I appreciate the discussion. The paper you sent is interesting. I agree it looks like for moderate values of K (on the order of 100-1000), RL models actually look a little worse at pass@k than their base models. So perhaps the right framing of your/Sutton's claim is: RL can upweight low-probability (p) but correct outputs, but there is a limit to how small p can be, and it is on the order of 1 in a 100 or 1 i…

Pass@128 is a lot. They were not easy.

> Discovery / creativity

I'm absolutely uninterested in the semantic discussions of what is a real discovery, what is creativity, what is intelligence, etc. I simply don't care. If it's useful great use it. If it's not great don't.

> How small p can be

All that depends on your sampling procedure. If you intentionally smooth the distribution out you can sample the smallest thing, but you pay for it with noise. Taken to an extreme, this is the monkeys typing on the keyboard argument.

It's a mathematical fact that RL cannot improve things it doesn't sample. In any learned distribution you pay a heavy cost by sampling far away from the mode. Most RL algos sample rollouts maybe with some smoothing but that's it. This is why external planners are necessary in order to sample something effectively un-sampleable in the base distribution. Simple example: tool use!

Sutton and everyone are simply calling for a focus on improving these external planners in the same way, as they also enable much better "continual" learning and so on.

> Erdos solution

The RL was what enabled such a huge trajectory to ever become efficiently sampleable in our lifetimes probably. You can do many useful things like this and more purely with the base model distribution.

In fact. Doing RL on user chats and so on especially from pair coding sessions are improving these models coding abilities by a lot making them even more reliable for SWE. In this regard, mode-seeking is a win.

> All sequences are technically in distribution

If it was truly improving 1 in million things systemically, then you wouldn't see base getting the same results given many samples. Albeit they are not erdos problems.

Could it be that at 1T scale, and for difficult problems specifically, grpo somehow filters through the noise and picks out the 1 in trillion? Extremely unlikely (you have your expected rollouts required to sample that, and then you have your sparse reward signal and no credit assignment on top of that...). But of course, only 2 companies in the world can do experiments with it, so there could be some unknown effect the rest of the world has not seen. Barring that, no.

Re: Rich Sutton on AI creativity and discovery

#140

Post by brokenmachine »

jason_oster wrote: post #125
antonvs wrote: post #114

Earlier quoted context omitted.

Commercial fusion energy is almost certainly a pipe dream. We already have good alternatives. Short of some unpredictable major breakthrough in physics on the level of something like cold fusion, the current activity in the fusion space will basically just serve to demonstrate definitively that it's not viable.

And then there's France, which gets about 70% of its electricity from nuclear energy.

Look up fission vs fusion.
Post reply on HN