Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Daydreaming

gwern.net

Post reply on HN
Print view View original

41–50 of 156 posts

Re: LLM Daydreaming

#41

Post by amelius »

Yizahi wrote: post #39
amelius wrote: post #23

Earlier quoted context omitted.

But what if the critic is just hard reality? If you ask an LLM to write a computer program, instead of criticizing it, you can run it and test it. If you ask an LLM to prove a theorem, let it write the proof in a formal logic language so it can be verified. Etcetera.

Generated code only works because "test" part (compile/validate/analyze etc.) is completely external and written before any mass-market LLMs. There is no such external validator for new theorems, books, pictures, text guides etc. You can't just run hard_reality.exe on a generated poem or a scientific paper to deem it "correct". It is only possible with programming languages, and even then not always.

Science is falsifiable by definition, and writing poems/books is not the kind of problem of interest here.

> There is no such external validator for new theorems

There are formal logic languages that will allow you to do this.

Re: LLM Daydreaming

#42

Post by zhangjunphy »

imtringued wrote: post #38
zhangjunphy wrote: post #10

I also hope we have something like this. But sadly, this is not going to work. The reason is this line from the article, which is so much harder that it looks: > and a critic model filters the results for genuinely valuable ideas. In fact, people have tryied this idea. And if you use a LLM or anything similar as the critic, the performance of the model actually degrades in this process. As the LLM tries too hard to s…

That didn't stop actor-critic from becoming one of the most popular deep RL methods.

True, and the successful ones usually require an external source of information. For AlphaGo, it is the simple algorithm which decide who is the winner of a game of Go. For GAN, it is the images labled by human. In these scenarios, the critic is the medium which transforms external information into gradient which optimized the actor, but not the direct source of that information.

Re: LLM Daydreaming

#43

Post by sneak »

Seems like an easy hypothesis to quickly smoke test with a couple hundred lines of script, a wikipedia index, and a few grand thrown at an API.

Re: LLM Daydreaming

#44

Post by Yizahi »

amelius wrote: post #41
Yizahi wrote: post #39

Earlier quoted context omitted.

Generated code only works because "test" part (compile/validate/analyze etc.) is completely external and written before any mass-market LLMs. There is no such external validator for new theorems, books, pictures, text guides etc. You can't just run hard_reality.exe on a generated poem or a scientific paper to deem it "correct". It is only possible with programming languages, and even then not always.

Science is falsifiable by definition, and writing poems/books is not the kind of problem of interest here. > There is no such external validator for new theorems There are formal logic languages that will allow you to do this.

Your proposed approach to science would result in the extremely tiny subset of math, probably theorems being proven by automation. And it is questionable if those theorems would be even useful. A good mathematician with CS experience can probably write a generator of new useless theorems, something along "are every sequential cube plus square of a number divisible by a root of seventh smallest prime multiplied by logn of than number plus blabla...". One can generate such theorrems and formally prove or disprove them, yes.

On the other hand any novel science usually requires deep and wide exploratory research, often involving hard or flawed experimentation or observation. One can train LLM on a PhD curriculum in astrophysics, then provide that LLM with API to some new observatory and instruct it to "go prove cosmological constant". And it will do so, but the result will be generated garbage because there is no formal way to prove such results. There is no formal way to prove why pharaohs decided to stop building pyramids, despite there being some decent theories. This is science too, you know. You can't formally prove that some gene sequence is responsible for trait X etc.

I would say a majority of science is not formally provable.

And lastly, you dismiss books/texts, but that is a huge chunk of intellectual and creative work of humans. Say you are an engineer and you have a CAD model with a list of parts and parameters for rocket for example. Now you need to write a guide for it. LLM can do that, it can generate guide-looking output. The issue is that there is no way to automatically proof it or find issues in it. And there are lots of items like that.

Re: LLM Daydreaming

#45

Post by Yizahi »

ashdksnndck wrote: post #8

I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it? First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for…

This is bordering conspiracy theory. Thousands of people are getting novel breakthroughs generated purely by LLM an not a single person discloses such result? Not even one of the countless LLM corporation engineers who depend on the billion dollar IV injections from deluded bankers just to continue surviving, and not one has bragged about LLM doing that revolution? Hard to believe.

Re: LLM Daydreaming

#46

Post by dr_dshiv »

Yes! I’ve been prototyping dreaming LLMs based on my downloaded history—and motivated by biomimetic design approaches. Just to surface ideas to myself again.

Re: LLM Daydreaming

#47

Post by amelius »

Yizahi wrote: post #44
amelius wrote: post #41

Earlier quoted context omitted.

Science is falsifiable by definition, and writing poems/books is not the kind of problem of interest here. > There is no such external validator for new theorems There are formal logic languages that will allow you to do this.

Your proposed approach to science would result in the extremely tiny subset of math, probably theorems being proven by automation. And it is questionable if those theorems would be even useful. A good mathematician with CS experience can probably write a generator of new useless theorems, something along "are every sequential cube plus square of a number divisible by a root of seventh smallest prime multiplied by log…

I think the problem here is that you assume the LLM has to operate isolated from the world, i.e. without interaction. If you put a human scientist in isolation, then you cannot have high expectations either.

Re: LLM Daydreaming

#48

Post by zhangjunphy »

amelius wrote: post #23
zhangjunphy wrote: post #10

I also hope we have something like this. But sadly, this is not going to work. The reason is this line from the article, which is so much harder that it looks: > and a critic model filters the results for genuinely valuable ideas. In fact, people have tryied this idea. And if you use a LLM or anything similar as the critic, the performance of the model actually degrades in this process. As the LLM tries too hard to s…

But what if the critic is just hard reality? If you ask an LLM to write a computer program, instead of criticizing it, you can run it and test it. If you ask an LLM to prove a theorem, let it write the proof in a formal logic language so it can be verified. Etcetera.

I think if we can have a good enough simulation of reality, and a fast one. Something like an accelerable minecraft with real world physics. Then this idea might actually work. But the hard reality we currenly could generate efficiently and feed into LLMs usually has a narrow scope. It feels liking teaching only textbook math to a kid for several years but nothing else. The LLM mostly overoptimize in these very specific fields, but the overall performance might even be worse.

Re: LLM Daydreaming

#49

Post by A_D_E_P_T »

> You are a creative synthesizer. Your task is to find deep, non-obvious, and potentially groundbreaking connections between the two following concepts. Do not state the obvious. Generate a hypothesis, a novel analogy, a potential research question, or a creative synthesis. Be speculative but ground your reasoning.

> Concept 1: {Chunk A} > Concept 2: {Chunk B}

In addition to the other criticisms mentioned by posters ITT, a problem I see is: What concepts do you feed it?

Obviously there's a problem with GIGO. If you don't pick the right concepts to begin with, you're not going to get a meaningful result. But, beyond that, human discovery (in mechanical engineering, at least,) tends to be massively interdisciplinary and serendipitous, so that many concepts are often involved, and many of those are necessarily non-obvious.

I guess you could come up with a biomimetics bot, but, besides that, I'm not so sure how well this concept would work as laid out above.

There's another issue in that LLMs tend to be extremely gullible, and swallow the scientific literature and University press releases verbatim and uncritically.

Re: LLM Daydreaming

#50

Post by yunohn »

Yizahi wrote: post #39
amelius wrote: post #23

Earlier quoted context omitted.

But what if the critic is just hard reality? If you ask an LLM to write a computer program, instead of criticizing it, you can run it and test it. If you ask an LLM to prove a theorem, let it write the proof in a formal logic language so it can be verified. Etcetera.

Generated code only works because "test" part (compile/validate/analyze etc.) is completely external and written before any mass-market LLMs. There is no such external validator for new theorems, books, pictures, text guides etc. You can't just run hard_reality.exe on a generated poem or a scientific paper to deem it "correct". It is only possible with programming languages, and even then not always.

IME, on a daily basis, Claude Code (supposed SoTA agent) constantly disables and bypasses tests and checks on my codebase - despite following clear prompting guidelines and all the /woo/ like ultrathink etc.
Post reply on HN