Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rich Sutton on AI creativity and discovery

twitter.com

Post reply on HN
Print view View original

121–130 of 141 posts

Re: Rich Sutton on AI creativity and discovery

#121

Post by highfrequency »

porridgeraisin wrote: post #83
highfrequency wrote: post #18

Unless I'm missing something, this argument seems to apply only to the original pretraining era (eg GPT 1-4). The post-training and reinforcement learning paradigms are clearly doing variation, evaluation and selective retention no?

RLVR still does not expand beyond the base distribution though, it only mode-seeks within it. i.e, evaluation, retention yes. variation or "planning" no. That is not to say you cannot use LLMs. Alpha evolve does exactly that. It uses an external simple evolutionary planner though. The overarching point he's making is that our planner is still "dumb" and we need to work on it. When you iteratively guide an LLM in clau…

> RLVR still does not expand beyond the base distribution though, it only mode-seeks within it.

Seems clearly false. Pretraining finds the mean/mode of the data distribution. RL can easily generate many samples around that mode, evaluate them on an external source of truth (eg compile the code and run it) and then selectively train on the good samples. This clearly can go beyond the initial data distribution.

Re: Rich Sutton on AI creativity and discovery

#122

Post by jason_oster »

doctoboggan wrote: post #40

> That is, I would say that creativity requires that the new things generated be Evaluated. Without evaluation, and retention of the best, there is nothing created. The novelty flickers into existence but, if its value is unrecognized, it flickers away and is lost. I really like the way he frames this here. I think a lot of people in the twitter comments (and maybe a few here) aren't reading past the introduction. He…

I agree in general. But making this claim with a narrowed definition of generative AI that doesn't match what is being used in practice is setting up a strawman.

Re: Rich Sutton on AI creativity and discovery

#123

Post by porridgeraisin »

highfrequency wrote: post #121
porridgeraisin wrote: post #83

Earlier quoted context omitted.

RLVR still does not expand beyond the base distribution though, it only mode-seeks within it. i.e, evaluation, retention yes. variation or "planning" no. That is not to say you cannot use LLMs. Alpha evolve does exactly that. It uses an external simple evolutionary planner though. The overarching point he's making is that our planner is still "dumb" and we need to work on it. When you iteratively guide an LLM in clau…

> RLVR still does not expand beyond the base distribution though, it only mode-seeks within it. Seems clearly false. Pretraining finds the mean/mode of the data distribution. RL can easily generate many samples around that mode, evaluate them on an external source of truth (eg compile the code and run it) and then selectively train on the good samples. This clearly can go beyond the initial data distribution.

by base distribution, I meant the base model's output distribution

Re: Rich Sutton on AI creativity and discovery

#124

Post by jason_oster »

oedemis wrote: post #63

why no one talks about claude shannon deep understanding of information theory and the consequences for compression and intelligence

Can't tell if you are spending time in the wrong circles or just shitposting. Here's one: https://youtu.be/l6DKRf-fAAM?si=6u54mry0TLgS868a

Re: Rich Sutton on AI creativity and discovery

#125

Post by jason_oster »

antonvs wrote: post #114
musebox35 wrote: post #56

Earlier quoted context omitted.

I understand the skepticism. I am worried about the implications of AI as well. The deeper issue at stake is that the depth of scientific knowledge has been increasing for a very long time. Now you get to have a PhD in esoteric subproblems and that slows down research especially if the discoveries require depth in multiple subdomains. Socially and economically training people in every combinatorial combination of sub…

Commercial fusion energy is almost certainly a pipe dream. We already have good alternatives. Short of some unpredictable major breakthrough in physics on the level of something like cold fusion, the current activity in the fusion space will basically just serve to demonstrate definitively that it's not viable.

And then there's France, which gets about 70% of its electricity from nuclear energy.

Re: Rich Sutton on AI creativity and discovery

#126

Post by antonvs »

jason_oster wrote: post #125
antonvs wrote: post #114

Earlier quoted context omitted.

Commercial fusion energy is almost certainly a pipe dream. We already have good alternatives. Short of some unpredictable major breakthrough in physics on the level of something like cold fusion, the current activity in the fusion space will basically just serve to demonstrate definitively that it's not viable.

And then there's France, which gets about 70% of its electricity from nuclear energy.

Not sure what you’re trying to say. France gets 0% of its energy from fusion, and the same is true of every country on Earth.

Re: Rich Sutton on AI creativity and discovery

#127

Post by Shitty-kitty »

artistonn wrote: post #67
Shitty-kitty wrote: post #30

One has to be very specific when throwing around words like "creative" when talking about A.I Can A.I create art. Well it can create something that's pleasing to our senses but art is ultimately about conveying human feelings and emotions. Even as humans, understanding art is not universal. "feelings and emotions" and therefore art, can be deeply tied to a particular groups shared beliefs and experiences. Can it be c…

> art is ultimately about conveying human feelings and emotions you made a small error, art is mostly about generating an emotion in the viewer/listener/.... not about transmitting an emotion of the creator the Wikipedia page on art starts with: > Art is a diverse range of cultural activity centered around works utilizing creative or imaginative talents, which are expected to evoke a worthwhile experience https://en.…

Using your provided definition, you come to a conclusion completely ignoring the "utilizing creative or imaginative talents" part.

Re: Rich Sutton on AI creativity and discovery

#128

Post by highfrequency »

porridgeraisin wrote: post #123
highfrequency wrote: post #121

Earlier quoted context omitted.

> RLVR still does not expand beyond the base distribution though, it only mode-seeks within it. Seems clearly false. Pretraining finds the mean/mode of the data distribution. RL can easily generate many samples around that mode, evaluate them on an external source of truth (eg compile the code and run it) and then selectively train on the good samples. This clearly can go beyond the initial data distribution.

by base distribution, I meant the base model's output distribution

The model’s distribution will certainly change from the base model’s output distribution during reinforcement learning, shifting toward outputs that score well on an external evaluation. This is very different from mode-seeking. Am I missing something?

Re: Rich Sutton on AI creativity and discovery

#129

Post by balazstorok »

YeGoblynQueenne wrote: post #105
balazstorok wrote: post #19

There seems to be a problem with how he poses the problems alphaGo and these GAI models face. AlphaGO is given a hard evaluation externally. It did not itself come up with it. When GAI models are given an external hard evaluation, they can also succeed in many different domains (that is one of the remarkable features, succeeding in many domains) ranging from simple programming tasks to frontier mathematics (disprovin…

>> (Insert the usual stuff like a plane being unable to fly like a bird or a submarine not swimming is totally irrelevant to it being useful). Just a brief reminder that planes have wings with airfoils just like birds and submarines have air tanks just like fish have swimming bladders. Some birds fly without flapping their wings much, too, e.g. albatrosses.

You are right in that. Propulsion is different in both cases. The point is, until you have not recreated the function, difficult to say which part is important in other implementations (~Marr levels).

Leonardo tried to create flapping wings.

Re: Rich Sutton on AI creativity and discovery

#130

Post by porridgeraisin »

highfrequency wrote: post #128
porridgeraisin wrote: post #123

Earlier quoted context omitted.

by base distribution, I meant the base model's output distribution

The model’s distribution will certainly change from the base model’s output distribution during reinforcement learning, shifting toward outputs that score well on an external evaluation. This is very different from mode-seeking. Am I missing something?

Mode-seeking is describing the way in which the distribution changes. RL is capable of picking out slightly lower probability trajectories and moving them toward the top of the distribution. However, exploration is fundamentally limited by the base policy itself. If a trajectory has near-zero probability under the original model, RLVR is unlikely to discover it because it must first be sampled before it can be rewarded. External search/planning methods such as MCTS or evolutionary search are useful precisely because they can explore candidate trajectories beyond what the policy would ordinarily generate. This is also not theoretical, GRPO style methods are shown to mostly improve `maj@k` and `pass@1` evals while not so much `pass@k` especially for high k, meaning it mostly sharpening the top of the distribution.

I'm not saying this makes it useless - it clearly helps for math and coding tasks. But the ceiling exists and that's what the original tweet was referring to. Alpha evolve also shows what lies beyond the ceiling, altho their planner was rudimentary.

Post reply on HN