Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Daydreaming

gwern.net

Post reply on HN
Print view View original

81–90 of 156 posts

Re: LLM Daydreaming

#81

Post by jerf »

amelius wrote: post #23
zhangjunphy wrote: post #10

I also hope we have something like this. But sadly, this is not going to work. The reason is this line from the article, which is so much harder that it looks: > and a critic model filters the results for genuinely valuable ideas. In fact, people have tryied this idea. And if you use a LLM or anything similar as the critic, the performance of the model actually degrades in this process. As the LLM tries too hard to s…

But what if the critic is just hard reality? If you ask an LLM to write a computer program, instead of criticizing it, you can run it and test it. If you ask an LLM to prove a theorem, let it write the proof in a formal logic language so it can be verified. Etcetera.

Those things are being done. Program testing is now off-the-shelf tech, and as for math proofs, see: https://www.geeky-gadgets.com/google-deepmind-alphaproof/

Re: LLM Daydreaming

#82

Post by HarHarVeryFunny »

vonneumannstan wrote: post #78
HarHarVeryFunny wrote: post #76

Earlier quoted context omitted.

It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, by using RL/etc to bias the generative process (according to reasoning data, good t…

>It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, but using RL/etc to bias the generative process (according to reasoning data, good…

Sure - but only if the unseen problem can be solved via the deductive/generative closure of the training data. And of course this type of "reasoning" is only as good as the RL pre-training it is based on - working well for closed domains like math where verification is easy, and not so well in the more general case.

Re: LLM Daydreaming

#83

Post by jacobr1 »

imiric wrote: post #66
danenania wrote: post #58

Earlier quoted context omitted.

Check out the DeepSeek paper. Research/benchmarks aside, try giving a somewhat hard programming task to Opus 4 with reasoning off vs. on. Similarly, try the same with o3 vs. o3-pro (o3-pro reasons for much longer). I'm not going to dig through my history for specific examples, but I do these kinds of comparisons occasionally when coding, and it's not unusual to have e.g. a bug that o3 can't figure out, but o3-pro can…

Huh, I wasn't aware that reasoning could be toggled. I use the OpenRouter API, and just saw that this is supported both via their web UI and API. I'm used to Sonnet 3.5 and 4 without reasoning, and their performance is roughly the same IME. I wouldn't trust comparing two different models, even from the same provider and family, since there could be many reasons for the performance to be different. Their system prompt…

It depends on the problem domain you have and the way you prompt things. Basically the reasoning is better, in cases where using the same model to critique itself in multiple turns would be better.

With code, for example, if a single shot without reasoning would have hallucinating a package or not conformed to the rest of the project style. Then you ask the llm check. Then ask it to revise itself to fix the issue. If the base model can do that - then turning on reasoning, basically allows it to self check for the self-correctable features.

When generating content, you can ask it to consider or produce intermediate deliverables like summaries of input documents that it then synthesizes into the whole. With reasoning on, it can do the intermediate steps and then use that.

The main advantage is that the system is autonomously figuring out a bunch of intermediate steps and working through it. Again no better than it probably could do with some guidance on multiple interactions - but that itself is a big productivity benefit. The second gen (or really 1.5 gen) reasoning models also seem to have been trained on enough reasoning traces that they are starting to know about additional factors to consider so the reasoning loop is tighter.

Re: LLM Daydreaming

#84

Post by jacobr1 »

Yizahi wrote: post #44
amelius wrote: post #41

Earlier quoted context omitted.

Science is falsifiable by definition, and writing poems/books is not the kind of problem of interest here. > There is no such external validator for new theorems There are formal logic languages that will allow you to do this.

Your proposed approach to science would result in the extremely tiny subset of math, probably theorems being proven by automation. And it is questionable if those theorems would be even useful. A good mathematician with CS experience can probably write a generator of new useless theorems, something along "are every sequential cube plus square of a number divisible by a root of seventh smallest prime multiplied by log…

> You can't formally prove that some gene sequence is responsible for trait X etc.

Maybe not formally in some kind of mathematical sense. But you certainly could have simulation models of protein synthesis, and maybe even higher order simulation of tissues and organs. You could also let the ai scientist verify the experimental hypothesis by giving access to robotic lab processes. In fact it seems we are going down both fronts right now.

Re: LLM Daydreaming

#85

Post by HarHarVeryFunny »

vonneumannstan wrote: post #77
HarHarVeryFunny wrote: post #67

> Despite impressive capabilities, large language models have yet to produce a genuine breakthrough. The puzzle is why. I don't see why this is remotely surprising. Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the…

> Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the training data - they are essentially built for recall, not creativity. This is just a completely base level of understanding of LLMs. How do you predict the next…

> This is just a completely base level of understanding of LLMs. How do you predict the next token with superhuman accuracy? Really think about how that is possible. If you think it's just stochastic parroting you are ngmi.

Yes, thank-you, I do understand how LLMs work. They learn a lot of generative rules from the training data, and will apply them in flexible fashion according to the context patterns they have learnt. You said stochastic parroting, not me.

However, we're not discussing whether LLMs can be superhuman at tasks where they had the necessary training - we're discussing whether they are capable of creativity (and presumably not just the trivially obvious case of being able to apply their generative rules in any order - deductive closure, not stochastic parroting in the dumbest sense of that expression).

Re: LLM Daydreaming

#86

Post by jacobr1 »

n4r9 wrote: post #21
jumploops wrote: post #12

How do you critique novelty? The models are currently trained on a static set of human “knowledge” — even if they “know” what novelty is, they aren’t necessarily incentivized to identify it. In my experience, LLMs currently struggle with new ideas, doubly true for the reasoning models with search. What makes novelty difficult, is that the ideas should be nonobvious (see: the patent system). For example, hallucinating…

It's a good question. A related question is: "what's an example of something undeniably novel?". Like if you ask an agent out of the blue to prove the Collatz conjecture, and it writes out a proof or counterexample. If that happens with LLMs then I'll be a lot more optimistic about the importance to AGI. Unfortunately, I suspect it will be a lot murkier than that - many of these big open questions will get chipped aw…

Good point. Look at patents. Few are truly novel in some exotic sense of "the whole idea is something never seen before." Most likely it is a combination of known factors applied in a new way, or incremental development improving on known techniques. In a banal sense, most LLM content generated is novel, in that the specific paragraphs might be unique combinations of words, even if the ideas are just slightly rearranged regurgitations.

So I strongly agree that, especially when are talking about the bulk of human discovery and invention, the incrementalism will be increasingly in striking distance of human/AI collaboration. Attribution of the novelty in these cases is going to be unclear, when the task is, simplified something like, "search for combinations of things, in this problem domain, that do the task better than some benchmark" be that drug discovery, maths, ai itself or whatever.

Re: LLM Daydreaming

#87

Post by yahoozoo »

I once asked ChatGPT to come up with a novel word that would return 0 Google search results. It came up with “vexlithic” which does indeed return 0 results, at least for me. I thought that was neat.

Re: LLM Daydreaming

#88

Post by js8 »

HarHarVeryFunny wrote: post #72
js8 wrote: post #63

I am not sure why tie this to any concrete AI technology such as LLMs. IMHO the biggest issue we have with AI right now is that we don't know how to philosophicaly formalize what we want. What is reasoning? I am trying to answer that for myself. Since every logic is expressible in untyped lambda calculus (as any computation is), you could have a system that just somehow generates terms and beta-reduces them. In even…

Reasoning is chained what-if prediction, together with exploration of alternatives (cf backtracking), and leans upon general curiosity/learning for impasse resolution (i.e. if you can't predict what-if, then have the curiosity to explore and find out). What the LLM companies are currently selling as "reasoning" is mostly RL-based pre-training whereby the model is encouraged to predict tokens (generate reasoning steps…

I am talking about reasoning in philosophical not logical sense. In your definition, you're assuming a logic in which reasoning happens, but when I am asking the question, I am not presuming any specific logic.

So how do you pick the logic in which to do reasoning? There are "good reasons" to use one logic over another.

LLMs probably learn some combination of logic rules (deduction rules in commonly used logics), but cannot guarantee they will be used consistently (i.e. choose a logic for the problem and stick to it). How do you accomplish that?

And even then reasoning is more than search. If you can reason, you should also be able to reason about more effective reasoning (for example better heuristics to cutting the search tree).

Re: LLM Daydreaming

#89

Post by awongh »

HarHarVeryFunny wrote: post #76
karmakaze wrote: post #71

Earlier quoted context omitted.

Yes LLMs choose probable sequences because they recognize similarity. Because of that, it can diverge from similarity to be creative: increase the temperature. What LLMs don't have is (good) taste—we need to build an artificial tongue and feed it as a prerequisite.

It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, by using RL/etc to bias the generative process (according to reasoning data, good t…

By volume how much of human speech / writing is pattern matching and how much of it is truly original cognition that would pass your bar of creativity? It is probably 90% rote pattern matching.

I don't think LLMs are AGI, but in most senses I don't think people give enough credit to their capabilities.

It's just ironic how human-like the flaws of the system are. (Hallucinations that are asserting untrue facts, just because they are plausible from a pattern matching POV)

Re: LLM Daydreaming

#90

Post by kajumix »

ashdksnndck wrote: post #8

I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it? First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for…

Most interesting novel ideas originate at the intersection of multiple disciplines. Profitable trades could be found in the biomedicine sector when the knowledge of biomedicine and finance are combined. That's where I see LLMs shining because they span disciplines way more than any human can. Once we figure out a way to have them combine ideas (similar to how Gwern is suggesting), there will be, I suspect, a flood of novel and interesting ideas, inconceivable with humans.
Post reply on HN