Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Daydreaming

gwern.net

Post reply on HN
Print view View original

21–30 of 156 posts

Re: LLM Daydreaming

#21

Post by n4r9 »

jumploops wrote: post #12

How do you critique novelty? The models are currently trained on a static set of human “knowledge” — even if they “know” what novelty is, they aren’t necessarily incentivized to identify it. In my experience, LLMs currently struggle with new ideas, doubly true for the reasoning models with search. What makes novelty difficult, is that the ideas should be nonobvious (see: the patent system). For example, hallucinating…

It's a good question. A related question is: "what's an example of something undeniably novel?". Like if you ask an agent out of the blue to prove the Collatz conjecture, and it writes out a proof or counterexample. If that happens with LLMs then I'll be a lot more optimistic about the importance to AGI. Unfortunately, I suspect it will be a lot murkier than that - many of these big open questions will get chipped away at by a combination of computational and human efforts, and it will be impossible to pinpoint where the "novelty" lies.

Re: LLM Daydreaming

#22

Post by pilooch »

AlphaEvolve and similar systems based on map-elites + DL/LLM + RL appears to be one of the promising paths.

Setting up the map-elites dimensions may still be problem-specific but this could be learnt unsupervisedly, at least partially.

The way I see LLMs is as a search-spqce within tokens that manipulate broad concepts within a complex and not so smooth manifold. These concepts can be refined within other spaces (pixel -space, physical spaces, ...)

Re: LLM Daydreaming

#23

Post by amelius »

zhangjunphy wrote: post #10

I also hope we have something like this. But sadly, this is not going to work. The reason is this line from the article, which is so much harder that it looks: > and a critic model filters the results for genuinely valuable ideas. In fact, people have tryied this idea. And if you use a LLM or anything similar as the critic, the performance of the model actually degrades in this process. As the LLM tries too hard to s…

But what if the critic is just hard reality? If you ask an LLM to write a computer program, instead of criticizing it, you can run it and test it. If you ask an LLM to prove a theorem, let it write the proof in a formal logic language so it can be verified. Etcetera.

Re: LLM Daydreaming

#24

Post by amelius »

khalic wrote: post #15

Ugh, again with the anthropomorphizing. LLMs didn't come up with anything new because _they don't have agency_ and _do not reason_... We're looking at our reflection and asking ourselves why it isn't moving when we don't

> anthropomorphizing

Gwern isn't doing that here. They say: "[LLMs] lack some fundamental aspects of human thought", and then investigates that.

Re: LLM Daydreaming

#25

Post by guelo »

In a recent talk [0] Francois Chollet made it sound like all the frontier models are doing Test-Time Adaptation, which I think is a similar concept to Dynamic evaluation that Gwern says is not being done. Apparently Test-Time Adaptation encompasses several techniques some of which modify model weights and some that don't, but they are all about on-the-fly learning.

[0] https://www.youtube.com/watch?v=5QcCeSsNRks&t=1542s

Re: LLM Daydreaming

#27

Post by LourensT »

Regardless of accusations of anthropomorphizing, continual thinking seems to be a precursor to any sense of agency, simply because agency requires something to be running.

Eventually LLM output degrades when most of the context is its own output. So should there also be an input stream of experience? The proverbial "staring out the window", fed into the model to keep it grounded and give hooks to go off?

Re: LLM Daydreaming

#28

Post by amelius »

Humans daydream about problems when they think a problem is interesting. Can an LLM know when a problem is interesting and thereby prune the daydream graph?

Re: LLM Daydreaming

#29

Post by MrScruff »

cantor_S_drug wrote: post #11
johnfn wrote: post #3

It's an interesting premise, but how many people - are capable of evaluating the LLM's output to the degree that they can identify truly unique insights - are prompting the LLM in such a way that it could produce truly unique insights I've prompted an LLM upwards of 1,000 times in the last month, but I doubt more than 10 of my prompts were sophisticated enough to even allow for a unique insight. (I spend a lot of tim…

> are capable of evaluating the LLM's output to the degree that they can identify truly unique insights I noticed one behaviour in myself. I heard about a particular topic, because it was a dominant opinion in the infosphere. Then LLMs confirmed that dominant opinion (because it was heavily represented in the training) and I stopped my search for alternative viewpoints. So in a sense, LLMs are turning out to be anoth…

Yes, it seems like LLMs are system one thinking taken to the extreme. Reasoning was supposed to introduce some actual logic but you only have to play with these models for a short while to see that the reasoning tokens are a very soft constraint on the models eventual output.

Infact, they're trained to please us and so in general aren't very good at pushing back. It's incredibly easy to 'beat' an LLM in an argument since they often just follow your line of reasoning (it's in the models context after all).

Re: LLM Daydreaming

#30

Post by _heimdall »

blueflow wrote: post #9

I have not yet seen AI doing a critical evaluation of data sources. AI willcontradict primary sources if the contradiction is more prevalent in the training data. Something about the whole approach is bugged. My pet peeve: "Unix System Resources" as explanation for the /usr directory is a term that did not exist until the turn of the millenium (rumor is that a c't journalist made it up in 1999), but AI will retcon it…

> Something about the whole approach is bugged.

The bug is that LLMs are fundamentally designed for natural language processing and prediction, not logic or reasoning.

We may get to actual AI eventually, but an LLM architecture either won't be involved at all or it will act as a part of the system mimicking the language center of a brain.

Post reply on HN