Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Daydreaming

gwern.net

Post reply on HN
Print view View original

71–80 of 156 posts

Re: LLM Daydreaming

#71

Post by karmakaze »

HarHarVeryFunny wrote: post #67

> Despite impressive capabilities, large language models have yet to produce a genuine breakthrough. The puzzle is why. I don't see why this is remotely surprising. Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the…

Yes LLMs choose probable sequences because they recognize similarity. Because of that, it can diverge from similarity to be creative: increase the temperature. What LLMs don't have is (good) taste—we need to build an artificial tongue and feed it as a prerequisite.

Re: LLM Daydreaming

#72

Post by HarHarVeryFunny »

js8 wrote: post #63

I am not sure why tie this to any concrete AI technology such as LLMs. IMHO the biggest issue we have with AI right now is that we don't know how to philosophicaly formalize what we want. What is reasoning? I am trying to answer that for myself. Since every logic is expressible in untyped lambda calculus (as any computation is), you could have a system that just somehow generates terms and beta-reduces them. In even…

Reasoning is chained what-if prediction, together with exploration of alternatives (cf backtracking), and leans upon general curiosity/learning for impasse resolution (i.e. if you can't predict what-if, then have the curiosity to explore and find out).

What the LLM companies are currently selling as "reasoning" is mostly RL-based pre-training whereby the model is encouraged to predict tokens (generate reasoning steps) according to similar "goals" seen in the RL training data. This isn't general case reasoning, but rather just "long horizon" prediction based on the training data. It helps exploit the training data, but isn't going to generate novelty outside of the deductive closure of the training data.

Re: LLM Daydreaming

#73

Post by grey-area »

karmakaze wrote: post #71
HarHarVeryFunny wrote: post #67

> Despite impressive capabilities, large language models have yet to produce a genuine breakthrough. The puzzle is why. I don't see why this is remotely surprising. Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the…

Yes LLMs choose probable sequences because they recognize similarity. Because of that, it can diverge from similarity to be creative: increase the temperature. What LLMs don't have is (good) taste—we need to build an artificial tongue and feed it as a prerequisite.

Well they also don't have understanding, a model of the world, and the ability to reason (no chain-of-thought created by AI companies is not reasoning), as well as having no taste.

So there is quite a lot missing.

Re: LLM Daydreaming

#74

Post by esafak »

Yizahi wrote: post #45
ashdksnndck wrote: post #8

I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it? First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for…

This is bordering conspiracy theory. Thousands of people are getting novel breakthroughs generated purely by LLM an not a single person discloses such result? Not even one of the countless LLM corporation engineers who depend on the billion dollar IV injections from deluded bankers just to continue surviving, and not one has bragged about LLM doing that revolution? Hard to believe.

Countless people are increasing their productivity and talking about it here ad nauseam. Even researchers are leaning on language models; e.g., https://mathstodon.xyz/@tao/114139125505827565

We haven't successfully resolved famous unsolved research problems through language models yet but one can imagine that they will solve increasingly challenging problems over time. And if it happens in the hands of a researcher rather than model's lab, one can also imagine that the researcher will take credit, so you will still have the same question.

Re: LLM Daydreaming

#75

Post by AIPedant »

esafak wrote: post #74
Yizahi wrote: post #45

Earlier quoted context omitted.

This is bordering conspiracy theory. Thousands of people are getting novel breakthroughs generated purely by LLM an not a single person discloses such result? Not even one of the countless LLM corporation engineers who depend on the billion dollar IV injections from deluded bankers just to continue surviving, and not one has bragged about LLM doing that revolution? Hard to believe.

Countless people are increasing their productivity and talking about it here ad nauseam. Even researchers are leaning on language models; e.g., https://mathstodon.xyz/@tao/114139125505827565 We haven't successfully resolved famous unsolved research problems through language models yet but one can imagine that they will solve increasingly challenging problems over time. And if it happens in the hands of a researcher r…

The actual posts totally undermine your point:

  My general sense is that for research-level mathematical tasks at least, current models fluctuate between "genuinely useful with only broad guidance from user" and "only useful after substantial detailed user guidance", with the most powerful models having a greater proportion of answers in the former category.  They seem to work particularly well for questions that are so standard that their answers can basically be found in existing sources such as Wikipedia or StackOverflow; but as one moves into increasingly obscure types of questions, the success rate tapers off (though in a somewhat gradual fashion), and the more user guidance (or higher compute resources) one needs to get the LLM output to a usable form. (2/2)

Re: LLM Daydreaming

#76

Post by HarHarVeryFunny »

karmakaze wrote: post #71
HarHarVeryFunny wrote: post #67

> Despite impressive capabilities, large language models have yet to produce a genuine breakthrough. The puzzle is why. I don't see why this is remotely surprising. Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the…

Yes LLMs choose probable sequences because they recognize similarity. Because of that, it can diverge from similarity to be creative: increase the temperature. What LLMs don't have is (good) taste—we need to build an artificial tongue and feed it as a prerequisite.

It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it.

Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, by using RL/etc to bias the generative process (according to reasoning data, good taste or whatever), but you can't move beyond the training data to be truly creative.

Re: LLM Daydreaming

#77

Post by vonneumannstan »

HarHarVeryFunny wrote: post #67

> Despite impressive capabilities, large language models have yet to produce a genuine breakthrough. The puzzle is why. I don't see why this is remotely surprising. Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the…

> Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the training data - they are essentially built for recall, not creativity.

This is just a completely base level of understanding of LLMs. How do you predict the next token with superhuman accuracy? Really think about how that is possible. If you think it's just stochastic parroting you are ngmi.

>large language models have yet to produce a genuine breakthrough. The puzzle is why. I think you should really update on the fact that world class researchers are surprised by this. They understand something you don't and that is that it's clear these models build robust world models and that text prompts act as probes into those world models. The surprising part is that despite these sophisticated world models we can't seem to get unique insights out which almost surely already exist in those models. Even if all the model is capable of is memorizing text then just the sheer volume it has memorized should yield unique insights, no human can ever hope to hold this much text in their memory and then make connections between it.

It's possible we just lack the prompt creativity to get these insights out but nevertheless there is something strange happening here.

Re: LLM Daydreaming

#78

Post by vonneumannstan »

HarHarVeryFunny wrote: post #76
karmakaze wrote: post #71

Earlier quoted context omitted.

Yes LLMs choose probable sequences because they recognize similarity. Because of that, it can diverge from similarity to be creative: increase the temperature. What LLMs don't have is (good) taste—we need to build an artificial tongue and feed it as a prerequisite.

It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, by using RL/etc to bias the generative process (according to reasoning data, good t…

>It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, but using RL/etc to bias the generative process (according to reasoning data, good taste or whatever), but you can't move beyond the training data to be truly creative.

It's clear these models can actually reason on unseen problems and if you don't believe that you aren't actually following the field.

Re: LLM Daydreaming

#79

Post by nico »

danielbln wrote: post #59
nico wrote: post #57

Earlier quoted context omitted.

> but I don’t commit a comment saying “wow, this clever bit came from the model!” The other day, Claude Code started adding a small signature to the commit messages it was preparing for me. It said something like “This commit was co-written with Claude Code” and a little robot emoji I wonder if that just happened by accident or if Anthropic is trying to do something like Apple with the “sent from my iPhone”

See https://docs.anthropic.com/en/docs/claude-code/settings#avai... , specifically `includeCoAuthoredBy`

Thank you. And I guess they are trying to do the Apple thing by making that option true by default

Re: LLM Daydreaming

#80

Post by vintagedave »

> Hypothesis: Day-Dreaming Loop

This mirrors something I have thought of too. I have read multiple theories of emerging consciousness, which touch on things from proprioception to the inner monologue (which not everyone has.)

My own theory is that -- avoiding the need for an awareness of a monologue -- a LLM loop that constantly takes input and lets it run, saving key summarised parts to memory that are then pulled back in when relevant, would be a very interesting system to speak to.

It would need two loops: the constant ongoing one, and then for interaction, one accessing memories from the first. The ongoing one would be aware of the conversation. I think it would be interesting to see what, via the memory system, would happen in terms of the conversation emitting elements from the loop.

My theory is that if we're likely to see emergent consciousness, it will come through ongoing awareness and memory.

Post reply on HN