Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Daydreaming

gwern.net

Post reply on HN
Print view View original

51–60 of 156 posts

Re: LLM Daydreaming

#51

Post by imiric »

yorwba wrote: post #20
imiric wrote: post #14

Earlier quoted context omitted.

Exactly. This not only affects a potential critic model, but the entire concept of a "reasoning" model is based on the same flawed idea—that the model can generate intermediate context to improve its final output. If that self-generated context contains hallucinations, baseless assumptions or doubt, the final output can only be an amalgamation of that. I've seen the "thinking" output arrive at a correct solution in t…

Reasoning models are trained from non-reasoning models of the same scale, and the training data is the output of the same model, filtered through a verifier. Generating intermediate context to improve the final output is not an idea that reasoning models are based on, but an outcome of the training process. Because empirically it does produce answers that pass the verifier more often if it generates the intermediate…

Thanks. I trust that you're more familiar with the internals than myself, so I stand corrected.

I'm only speaking from personal usage experience, and don't trust benchmarks since they are often gamed, but if this process produces objectively better results that aren't achieved by scaling up alone, then that's a good thing.

Re: LLM Daydreaming

#52

Post by imiric »

danenania wrote: post #34
imiric wrote: post #14

Earlier quoted context omitted.

Exactly. This not only affects a potential critic model, but the entire concept of a "reasoning" model is based on the same flawed idea—that the model can generate intermediate context to improve its final output. If that self-generated context contains hallucinations, baseless assumptions or doubt, the final output can only be an amalgamation of that. I've seen the "thinking" output arrive at a correct solution in t…

> The reason why "reasoning" models tend to perform better is simply due to larger scale and better training data. Except that we can try the exact same pre-trained model with reasoning enabled vs. disabled and empirically observe that reasoning produces better, more accurate results.

I'm curious: can you link to any tests that prove this?

I don't trust most benchmarks, but if this can be easily confirmed by an apples-to-apples comparison, then I would be inclined to believe it.

Re: LLM Daydreaming

#53

Post by dpoloncsak »

zhangjunphy wrote: post #48
amelius wrote: post #23

Earlier quoted context omitted.

But what if the critic is just hard reality? If you ask an LLM to write a computer program, instead of criticizing it, you can run it and test it. If you ask an LLM to prove a theorem, let it write the proof in a formal logic language so it can be verified. Etcetera.

I think if we can have a good enough simulation of reality, and a fast one. Something like an accelerable minecraft with real world physics. Then this idea might actually work. But the hard reality we currenly could generate efficiently and feed into LLMs usually has a narrow scope. It feels liking teaching only textbook math to a kid for several years but nothing else. The LLM mostly overoptimize in these very speci…

Its gotta be G-Mod

Re: LLM Daydreaming

#54

Post by RALaBarge »

yorwba wrote: post #17
khalic wrote: post #15

Ugh, again with the anthropomorphizing. LLMs didn't come up with anything new because _they don't have agency_ and _do not reason_... We're looking at our reflection and asking ourselves why it isn't moving when we don't

If you look at your reflection in water, it may very well move even though you don't. Similarly, you don't need agency or reasoning to create something new, random selection from a large number of combinations is enough, correct horse battery staple. Of course random new things are typically bad. The article is essentially proposing to generate lots of them anyway and try to filter for only the best ones.

I agree that brute forcing is a method and how nature does it. The problem would still be the same, how would it or other LLMs know if the idea is novel and interesting?

Given access to unlimited data, LLMs likely could spot novel trends that we cant but still cant judge the value of creating something unique that it has never encountered before.

Re: LLM Daydreaming

#55

Post by washadjeffmad »

bonoboTP wrote: post #16
aredox wrote: post #6

Oh, in the middle of "AI is PhD-level" propaganda (just check Google News to see this is not a strawman argument), some people finally admit in passing "no LLM has ever made a breakthrough". (See original argument: https://nitter.net/dwarkesh_sp/status/1727004083113128327 )

I agree there's an equivocation going on for "PhD level" between "so smart, it could get a PhD" (as in come up with and publish new research and defend its own thesis) and "it can solve quizzes at the level that PhDs can".

Services that make this claim are paying people with PhDs to ask their models questions and then provide feedback on the responses with detailed reasoning.

Re: LLM Daydreaming

#56

Post by RALaBarge »

RALaBarge wrote: post #54
yorwba wrote: post #17

Earlier quoted context omitted.

If you look at your reflection in water, it may very well move even though you don't. Similarly, you don't need agency or reasoning to create something new, random selection from a large number of combinations is enough, correct horse battery staple. Of course random new things are typically bad. The article is essentially proposing to generate lots of them anyway and try to filter for only the best ones.

I agree that brute forcing is a method and how nature does it. The problem would still be the same, how would it or other LLMs know if the idea is novel and interesting? Given access to unlimited data, LLMs likely could spot novel trends that we cant but still cant judge the value of creating something unique that it has never encountered before.

Yet.

Re: LLM Daydreaming

#57

Post by nico »

ashdksnndck wrote: post #8

I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it? First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for…

> but I don’t commit a comment saying “wow, this clever bit came from the model!”

The other day, Claude Code started adding a small signature to the commit messages it was preparing for me. It said something like “This commit was co-written with Claude Code” and a little robot emoji

I wonder if that just happened by accident or if Anthropic is trying to do something like Apple with the “sent from my iPhone”

Re: LLM Daydreaming

#58

Post by danenania »

imiric wrote: post #52
danenania wrote: post #34

Earlier quoted context omitted.

> The reason why "reasoning" models tend to perform better is simply due to larger scale and better training data. Except that we can try the exact same pre-trained model with reasoning enabled vs. disabled and empirically observe that reasoning produces better, more accurate results.

I'm curious: can you link to any tests that prove this? I don't trust most benchmarks, but if this can be easily confirmed by an apples-to-apples comparison, then I would be inclined to believe it.

Check out the DeepSeek paper.

Research/benchmarks aside, try giving a somewhat hard programming task to Opus 4 with reasoning off vs. on. Similarly, try the same with o3 vs. o3-pro (o3-pro reasons for much longer).

I'm not going to dig through my history for specific examples, but I do these kinds of comparisons occasionally when coding, and it's not unusual to have e.g. a bug that o3 can't figure out, but o3-pro can. I think this is widely accepted by engineers using LLMs to help them code; it's not controversial.

Re: LLM Daydreaming

#59

Post by danielbln »

nico wrote: post #57
ashdksnndck wrote: post #8

I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it? First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for…

> but I don’t commit a comment saying “wow, this clever bit came from the model!” The other day, Claude Code started adding a small signature to the commit messages it was preparing for me. It said something like “This commit was co-written with Claude Code” and a little robot emoji I wonder if that just happened by accident or if Anthropic is trying to do something like Apple with the “sent from my iPhone”

See https://docs.anthropic.com/en/docs/claude-code/settings#avai..., specifically `includeCoAuthoredBy`

Re: LLM Daydreaming

#60

Post by sartak »

From The Metamorphosis of Prime Intellect (1994):

> Among Prime Intellect's four thousand six hundred and twelve interlocking programs was one Lawrence called the RANDOM_IMAGINATION_ENGINE. Its sole purpose was to prowl for new associations that might fit somewhere in an empty area of the GAT. Most of these were rejected because they were useless, unworkable, had a low priority, or just didn't make sense. But now the RANDOM_IMAGINATION_ENGINE made a critical connection, one which Lawrence had been expecting it to make [...]

> Deep within one of the billions of copies of Prime Intellect, one copy of the Random_Imagination_Engine connected two thoughts and found the result good. That thought found its way to conscious awareness, and because the thought was so good it was passed through a network of Prime Intellects, copy after copy, until it reached the copy which had arbitrarily been assigned the duty of making major decisions -- the copy which reported directly to Lawrence. [...]

> "I've had an idea for rearranging my software, and I'd like to know what you think."

> At that Lawrence felt his blood run cold. He hardly understood how things were working as it was; the last thing he needed was more changes. "Yes?"

Post reply on HN