Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Daydreaming

gwern.net

Post reply on HN
Print view View original

91–100 of 156 posts

Re: LLM Daydreaming

#91

Post by js8 »

vonneumannstan wrote: post #78
HarHarVeryFunny wrote: post #76

Earlier quoted context omitted.

It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, by using RL/etc to bias the generative process (according to reasoning data, good t…

>It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, but using RL/etc to bias the generative process (according to reasoning data, good…

Both can be true (and that's why I downvoted you in the other comment, for presenting this as a dichotomy), LLMs can reason and yet "stochastically parrot" the training data.

For example, LLM might learn a rule that sentences that are similar to "A is given. From A follows B.", are followed by statement "Therefore, B". This is modus ponens. LLM can apply this rule to wide variety of A and B, producing novel statements. Yet, these statements are still the statistically probable ones.

I think the problem is, when people say "AI should produce something novel" (or "are producing", depending whether they advocate or dismiss), they are not very clear what the "novel" actually means. Mathematically, it's very easy to produce a never-before-seen theorem; but is it interesting? Probably not.

Re: LLM Daydreaming

#92

Post by fragmede »

HarHarVeryFunny wrote: post #67

> Despite impressive capabilities, large language models have yet to produce a genuine breakthrough. The puzzle is why. I don't see why this is remotely surprising. Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the…

Define creativity. Three things LLMs can do is write song lyrics, poems, and jokes, all of which require some level of what we think of as human creativity. Of course detractors will say LLM versions of those three aren't very good, and they may even be right, but a twelve year old child coming up with the same would be seen as creative, even if they didn't get significant recognition for it.

Re: LLM Daydreaming

#93

Post by Yizahi »

jacobr1 wrote: post #84
Yizahi wrote: post #44

Earlier quoted context omitted.

Your proposed approach to science would result in the extremely tiny subset of math, probably theorems being proven by automation. And it is questionable if those theorems would be even useful. A good mathematician with CS experience can probably write a generator of new useless theorems, something along "are every sequential cube plus square of a number divisible by a root of seventh smallest prime multiplied by log…

> You can't formally prove that some gene sequence is responsible for trait X etc. Maybe not formally in some kind of mathematical sense. But you certainly could have simulation models of protein synthesis, and maybe even higher order simulation of tissues and organs. You could also let the ai scientist verify the experimental hypothesis by giving access to robotic lab processes. In fact it seems we are going down bo…

Nobody argues that LLMs aren't useful for some bulk processing of billion datapoints or looking for obscure correlations in the unedited data. But the premise of the Gwern's article is that to be considered thinking, LLM must initiate such search on it's own and arrive to a novel conclusion on it's own.

Basically if:

A) Scientist has an idea > triggers LLM program to sift through a ton of data > LLM print out correlation results > scientist read them and proves/disproves an idea. In this case, while LLM did a bulk of work here, it did not arrive at a breakthrough on its own.

B) LLM is idling > then LLM triggers some API to get some specific set of data > LLM correlates results > LLM prints out a complete hypothesis with proof (or disproves it). In this case we can say that LLM did a breakthrough.

Re: LLM Daydreaming

#94

Post by HarHarVeryFunny »

js8 wrote: post #88
HarHarVeryFunny wrote: post #72

Earlier quoted context omitted.

Reasoning is chained what-if prediction, together with exploration of alternatives (cf backtracking), and leans upon general curiosity/learning for impasse resolution (i.e. if you can't predict what-if, then have the curiosity to explore and find out). What the LLM companies are currently selling as "reasoning" is mostly RL-based pre-training whereby the model is encouraged to predict tokens (generate reasoning steps…

I am talking about reasoning in philosophical not logical sense. In your definition, you're assuming a logic in which reasoning happens, but when I am asking the question, I am not presuming any specific logic. So how do you pick the logic in which to do reasoning? There are "good reasons" to use one logic over another. LLMs probably learn some combination of logic rules (deduction rules in commonly used logics), but…

OK, so maybe we're talking somewhat at cross purposes.

I was talking about the process/mechanism of reasoning - how do our brains appear to implement the capability that we refer to as "reasoning", and by extension how could an AI do the same by implementing the same mechanisms.

If we accept prediction (i.e use of past experience) as the mechanistic basis of reasoning, then choice of logic doesn't really come into it - it's more just a matter of your past experience and what you have learnt. What predictive rules/patterns have you learnt, both in terms of a corpus of "knowledge" you can bring to bear, but also in terms of experience with the particular problem domain - what have you learnt (i.e. what solution steps can you predict) about trying to reason about any given domain/goal ?

In terms of consistent use of logic, and sticking to it, one of the areas where LLMs are lacking is in not having any working memory other than their own re-consumed output, as well as an inability to learn beyond pre-training. With both of these capabilities an AI could maintain a focus (working memory) on the problem at hand (vs suffer from "context rot") and learn consistent, or phased/whatever, logic that has been successful in the past at solving similar problems (i.e predicting actions that will lead to solution).

Re: LLM Daydreaming

#95

Post by Yizahi »

therealpygon wrote: post #62
ashdksnndck wrote: post #8

I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it? First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for…

It is hard to accept as a premise because the premise is questionable from the beginning. Google already reported several breakthroughs as a direct result of AI, using processes that almost certainly include LLMs, including a new solution in math, improved chip designs, etc. DeepMind has AI that predicted millions of protein folds which are already being used in drugs among many other things they do, though yes, not…

You are contradicting yourself. Either LLM programs can do breakthrough on their own, or they don't have that capacity in a meaningful way based on how they work.

Re: LLM Daydreaming

#96

Post by Yizahi »

esafak wrote: post #74
Yizahi wrote: post #45

Earlier quoted context omitted.

This is bordering conspiracy theory. Thousands of people are getting novel breakthroughs generated purely by LLM an not a single person discloses such result? Not even one of the countless LLM corporation engineers who depend on the billion dollar IV injections from deluded bankers just to continue surviving, and not one has bragged about LLM doing that revolution? Hard to believe.

Countless people are increasing their productivity and talking about it here ad nauseam. Even researchers are leaning on language models; e.g., https://mathstodon.xyz/@tao/114139125505827565 We haven't successfully resolved famous unsolved research problems through language models yet but one can imagine that they will solve increasingly challenging problems over time. And if it happens in the hands of a researcher r…

Increasing productivity is nice and commendable, but it is NOT an LLM making a breakthrough on its own, which is the topic of the Gwern's article.

Re: LLM Daydreaming

#97

Post by Yizahi »

amelius wrote: post #47
Yizahi wrote: post #44

Earlier quoted context omitted.

Your proposed approach to science would result in the extremely tiny subset of math, probably theorems being proven by automation. And it is questionable if those theorems would be even useful. A good mathematician with CS experience can probably write a generator of new useless theorems, something along "are every sequential cube plus square of a number divisible by a root of seventh smallest prime multiplied by log…

I think the problem here is that you assume the LLM has to operate isolated from the world, i.e. without interaction. If you put a human scientist in isolation, then you cannot have high expectations either.

I assume not that LLM would be isolated, I assume that LLM would be incapable of interacting in any meaningful way on its own (i.e. not triggered by direct input from a programmer).

Re: LLM Daydreaming

#98

Post by js8 »

therealpygon wrote: post #62
ashdksnndck wrote: post #8

I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it? First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for…

It is hard to accept as a premise because the premise is questionable from the beginning. Google already reported several breakthroughs as a direct result of AI, using processes that almost certainly include LLMs, including a new solution in math, improved chip designs, etc. DeepMind has AI that predicted millions of protein folds which are already being used in drugs among many other things they do, though yes, not…

I would say that real breakthrough was training NNs as a way to create practical approximators for very complex functions over some kind of many-valued logics. Why they work so well in practice we still don't fully theoretically understand (in the sense we don't know what kind of underlying logic best models what we want from these systems). The LLMs (and application to natural language) are just a consequence of that.

Re: LLM Daydreaming

#99

Post by HarHarVeryFunny »

fragmede wrote: post #92
HarHarVeryFunny wrote: post #67

> Despite impressive capabilities, large language models have yet to produce a genuine breakthrough. The puzzle is why. I don't see why this is remotely surprising. Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the…

Define creativity. Three things LLMs can do is write song lyrics, poems, and jokes, all of which require some level of what we think of as human creativity. Of course detractors will say LLM versions of those three aren't very good, and they may even be right, but a twelve year old child coming up with the same would be seen as creative, even if they didn't get significant recognition for it.

Sure, but the author of TFA is well versed in LLMs and so is addressing something different. Novelty isn't the same as creativity, especially when limited to generating based on a fixed repertoire of moves.

The term "deductive closure" has been used to describe what LLMs are capable of, and therefore what they are not capable of. They can generate novelty (e.g. new poem) by applying the rules they have learnt in novel ways, but are ultimately restricted by their fixed weights and what was present in the training data, as well as being biased to predict rather than learn (which they anyways can't!) and explore.

An LLM may do a superhuman job of applying what it "knows" to create solutions to novel goals (be that a math olympiad problem, or some type of "creative" output that has been requested, such as a poem), but is unlikely to create a whole new field of math that wasn't hinted at in the training data because it is biased to predict, and anyways doesn't have the ability to learn that would allow it to build a new theory from the ground up one step at a time. Note (for anyone who might claim otherwise) that "in-context learning" is really a misnomer - it's not about learning but rather about using data that is only present in-context rather than having been in the training set.

Re: LLM Daydreaming

#100

Post by dingnuts »

awongh wrote: post #89
HarHarVeryFunny wrote: post #76

Earlier quoted context omitted.

It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, by using RL/etc to bias the generative process (according to reasoning data, good t…

By volume how much of human speech / writing is pattern matching and how much of it is truly original cognition that would pass your bar of creativity? It is probably 90% rote pattern matching. I don't think LLMs are AGI, but in most senses I don't think people give enough credit to their capabilities. It's just ironic how human-like the flaws of the system are. (Hallucinations that are asserting untrue facts, just b…

My intuition is opposite yours; due to the insane complexity of the real world nearly 90% of situations are novel and require creativity

OK now we're at an impasse until someone can measure this

Post reply on HN