Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Daydreaming

gwern.net

Post reply on HN
Print view View original

121–130 of 156 posts

Re: LLM Daydreaming

#121

Post by PaulHoule »

ashdksnndck wrote: post #8

I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it? First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for…

Almost certainly an LLM has, in response to a prompt and through sheer luck, spat out the kernel of an idea that a super-human centaur of the year 2125 would see as groundbreaking that hasn't been recognized as such.

We have a thin conception of genius that can be challenged by Edison's "1% inspiration, 99% perspiration" or the process of getting a PhD were you might spend 7 years getting to the point where you can start adding new knowledge and then take another 7 years to really hit your stride.

I have a friend who is 50-something and disabled with some mental illness, he thinks he has ADHD. We had a conversation recently where he repeatedly expressed his fantasy that he could show up somewhere with his unique perspective and sprinkle some pixie dust on their problems and be rewarded for it. I found it exhausting. When I would hear his ideas, or if I hear any idea, I immediately think "how would we turn this into a product and sell it?" or "write a paper about it?" or "convince people of it?" and he would have no part of it and think that operationalizing or advocating for that was uninteresting and that somebody else would do all that work and my answer is -- they might, but not without the advocacy.

And it comes down to that.

If an LLM were to come up with a groundbreaking idea and be recognized as having a groundbreaking idea it would have to do a sustained amount of work, say at least 2 person × years equivalent to win people over. And they aren't anywhere near equipped to do that, nobody is going to pay the power bill to do that, and if you were paying the power bill you'd probably have to pay the power bill for a million of them to go off in the wrong direction.

Re: LLM Daydreaming

#122

Post by andoando »

awongh wrote: post #89
HarHarVeryFunny wrote: post #76

Earlier quoted context omitted.

It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, by using RL/etc to bias the generative process (according to reasoning data, good t…

By volume how much of human speech / writing is pattern matching and how much of it is truly original cognition that would pass your bar of creativity? It is probably 90% rote pattern matching. I don't think LLMs are AGI, but in most senses I don't think people give enough credit to their capabilities. It's just ironic how human-like the flaws of the system are. (Hallucinations that are asserting untrue facts, just b…

What is the distinction between "pattern matching" and "original cognition" exactly?

All human ideas are a combination of previously seen ideas. If you disagree, come up with a truly new conception which is not. -- Badly quoted David hume

Re: LLM Daydreaming

#123

Post by zbyforgotp »

jumploops wrote: post #12

How do you critique novelty? The models are currently trained on a static set of human “knowledge” — even if they “know” what novelty is, they aren’t necessarily incentivized to identify it. In my experience, LLMs currently struggle with new ideas, doubly true for the reasoning models with search. What makes novelty difficult, is that the ideas should be nonobvious (see: the patent system). For example, hallucinating…

I think our minds don’t use novelty - but salience and it also might be easier to implement.

Re: LLM Daydreaming

#124

Post by suddenlybananas »

preciousoo wrote: post #110
suddenlybananas wrote: post #105

Earlier quoted context omitted.

The fact some people don't succeed doesn't show that humans operate by brute force. To claim humans reason and invent by brute force is patently absurd.

It’s an absurd statement because you are human and are aware of how research works on an individual level. Take yourself outside of that, and imagine you invented earth, added an ecosystem, and some humans. Wheels were invented ~6k years ago, and “humans” have existed for ~40-300k years. We can do the same for other technologies. As a group, we are incredibly inefficient, and an outside observer would see our efforts…

[flagged]

Re: LLM Daydreaming

#125

Post by jsbg »

zhangjunphy wrote: post #10

I also hope we have something like this. But sadly, this is not going to work. The reason is this line from the article, which is so much harder that it looks: > and a critic model filters the results for genuinely valuable ideas. In fact, people have tryied this idea. And if you use a LLM or anything similar as the critic, the performance of the model actually degrades in this process. As the LLM tries too hard to s…

> the LLM tries too hard to satisfy the critic

The LLM doesn't have to know about the critic though. It can just output things and the critic is a second process that filters the output for the end user.

Re: LLM Daydreaming

#126

Post by awongh »

HarHarVeryFunny wrote: post #107
awongh wrote: post #89

Earlier quoted context omitted.

By volume how much of human speech / writing is pattern matching and how much of it is truly original cognition that would pass your bar of creativity? It is probably 90% rote pattern matching. I don't think LLMs are AGI, but in most senses I don't think people give enough credit to their capabilities. It's just ironic how human-like the flaws of the system are. (Hallucinations that are asserting untrue facts, just b…

> It's just ironic how human-like the flaws of the system are. (Hallucinations that are asserting untrue facts, just because they are plausible from a pattern matching POV) I think most human mistakes are different - not applying a lot of complex logic to come to an incorrect deduction/guess (= LLM hallucination), but rather just shallow recall/guess. e.g. An LLM would guess/hallucinate a capital city by using rules…

The reason the LLMs are of any use to anyone right now (and real people are using them for real things right now- see the millions of ChatGPT users) is because the qualitative difference between text created by a real human using a guessing heuristic vs. an LLM using statistics is qualitatively the same. Even for things that some subjectively deem "creative".

The entropy of communication also makes it so that we mostly won't ever know when a person is guessing or if they think they're telling the truth. In that sense it makes less difference to the receiver of the information what the intent was- even if it came from a human, that human's guessing/ BS level is still unknown to you, the recipient.

This difference will continue to get smaller and more imperceptible. When it will stop changing or at what rate it will change is anyone's guess.

Re: LLM Daydreaming

#127

Post by awongh »

vonneumannstan wrote: post #77
HarHarVeryFunny wrote: post #67

> Despite impressive capabilities, large language models have yet to produce a genuine breakthrough. The puzzle is why. I don't see why this is remotely surprising. Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the…

> Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the training data - they are essentially built for recall, not creativity. This is just a completely base level of understanding of LLMs. How do you predict the next…

Not just prompting, it also could be we haven't done the right kind of RLHF for these kinds of outputs?

Re: LLM Daydreaming

#128

Post by heyjamesknight »

awongh wrote: post #89
HarHarVeryFunny wrote: post #76

Earlier quoted context omitted.

It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, by using RL/etc to bias the generative process (according to reasoning data, good t…

By volume how much of human speech / writing is pattern matching and how much of it is truly original cognition that would pass your bar of creativity? It is probably 90% rote pattern matching. I don't think LLMs are AGI, but in most senses I don't think people give enough credit to their capabilities. It's just ironic how human-like the flaws of the system are. (Hallucinations that are asserting untrue facts, just b…

The topic of conversation is not "human speech/writing" but "human creativity." There's no dispute that LLMs can create novel pieces of textual output. But there is no evidence that they can produce novel ideas. To assume they can is to adopt a purely rationalist approach to epistemology and cognition. Plato, Aquinas, and Kant would all fervently disagree with that approach.

Re: LLM Daydreaming

#129

Post by morsch »

nico wrote: post #57
ashdksnndck wrote: post #8

I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it? First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for…

> but I don’t commit a comment saying “wow, this clever bit came from the model!” The other day, Claude Code started adding a small signature to the commit messages it was preparing for me. It said something like “This commit was co-written with Claude Code” and a little robot emoji I wonder if that just happened by accident or if Anthropic is trying to do something like Apple with the “sent from my iPhone”

Aider does the same thing (and has a similar setting). I tend to squash the AI commits and remove it that way, though I suppose a flag indicating the degree of AI authorship could be useful.

Re: LLM Daydreaming

#130

Post by drdaeman »

suddenlybananas wrote: post #105
starlust2 wrote: post #101

Earlier quoted context omitted.

> through brute force The same is true of humanity in aggregate. We attribute discoveries to an individual or group of researchers but to claim humans are efficient at novel research is a form of survivorship bias. We ignore the numerous researchers who failed to achieve the same discoveries.

The fact some people don't succeed doesn't show that humans operate by brute force. To claim humans reason and invent by brute force is patently absurd.

Does “brute force” allow for heuristics and direction?

If it doesn’t (“brute” as opposite of “smart”, just dumb iteration to exhaustion) then you’re right, of course.

But if it does, then I’m not sure it’s patently absurd - novel ideas could be merely a matter of chance of having all the precursors together at the right time, a stochastic process. And it scales well, bearing at least some resemblance to brute force approaches - although the term is not entirely great (something around “stochastic”, “trial-and-error”, and “heuristic” is probably a better term).

Post reply on HN