Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A sleep-like consolidation mechanism for LLMs

arxiv.org

Post reply on HN
Print view View original

141–150 of 155 posts

Re: A sleep-like consolidation mechanism for LLMs

#141

Post by thunderbird120 »

soulofmischief wrote: post #137
thunderbird120 wrote: post #18

The idea of periodically stopping to write blocks of recent context into a fast-weight state is interesting, but I think it liked it better when E2E-TTT[1] did it. It's a more flexible and elegant continuous learning approach. Essentially it goes "You know how your model can remember its training data? Well, what if you treated its recent context like more training data and updated (some of) the weights using (mostly…

Each model needs to be a separate copy, or at least have those particular weights be interchangeable, for every single user. Remember Microsoft Tay. https://en.wikipedia.org/wiki/Tay_(chatbot)#Initial_release

Yes, since the weights being updated are a small subset of the overall total it's manageable. Just like how each separate conversation currently requires you to store a separate KV cache, you'd need to store the fast weights separately. Both KV cache and fast weight content stores have to be conversation specific, so just setting a bit of extra RAM aside for "memory" isn't really a new ask, just a different format for an old problem.

Re: A sleep-like consolidation mechanism for LLMs

#143

Post by fc417fc802 »

DonHopkins wrote: post #34
rahen wrote: post #23

That's an idea I had a few months ago: after going through a compaction once the KV cache is nearing capacity, accumulate this knowledge into a dataset to fine-tune a LoRA during offline hours. This would create a three-layer memory system: - Stable long-term memory (initial base weights) - Mid-term memory built from the compactions and replay buffers - Short-term memory (KV cache) Sleeping would just be a fancy term…

It's a network of computers with GPUs, so there's no reason it can't sleep at the same time it's awake. Just a continuous "sleeping" process going on in the background, incrementally updating the model. No need for the "thinking" process to be "unconscious" while the "sleeping" process runs. Anthropomorphism confuses everything. There's no such thing as "offline hours" because the Earth is a sphere and the United Sta…

> the Earth is a sphere and the United States is not the center of the universe.

Felt like stating the obvious there? Greenwich being the center of everything after all.

Re: A sleep-like consolidation mechanism for LLMs

#144

Post by jjk166 »

fc417fc802 wrote: post #139
Earw0rm wrote: post #115

Earlier quoted context omitted.

Probably because it evolved very early (like before bilateral symmetry, multi layer body cavity, or kidneys early... maybe even before multicellular animals early) and so has been incorporated as an essential pillar into multiple processes layered on top of that fundamental architecture.

You've merely stated observations about the context and the process that led to it. That doesn't in any way answer the question of what it's actually doing that's so essential.

It is not necessary to answer the question of what it's doing that is essential to know a subset of what it is doing.

Re: A sleep-like consolidation mechanism for LLMs

#145

Post by fc417fc802 »

jjk166 wrote: post #144
fc417fc802 wrote: post #139

Earlier quoted context omitted.

You've merely stated observations about the context and the process that led to it. That doesn't in any way answer the question of what it's actually doing that's so essential.

It is not necessary to answer the question of what it's doing that is essential to know a subset of what it is doing.

I don't think your response is coherent.

The question was "why is it needed". In context the meaning is clearly to ask what it's doing that's essential and (it follows) why those things are essential.

The subsequent response did not (as you suggest) articulate some subset of nonessential things done during sleep. Rather it rattled off plausible (and widely understood) aspects of the process that could have led to the current situation. Even if it had listed concrete activities that would still not have made for a meaningful answer.

The first clue that something is wrong should be that the linked article is recent and prominent. Thus short a brand new groundbreaking development we can be reasonably certain that a random commenter on the internet will not be sensibly rebutting the claims (and certainly not in the span of ~2 sentences).

Re: A sleep-like consolidation mechanism for LLMs

#146

Post by DoctorOetker »

rendx wrote: post #90
adastra22 wrote: post #33

Earlier quoted context omitted.

There is a lot that is known about sleep. We don't know everything and there are large gaps in our knowledge, but there is also a lot that we do know. And this research explicitly tried to emulate the things we know that sleep does do. Calling it "sleep" is warranted, imho.

"Despite myriad studies, there is still no consensus on why sleep is needed for survival." https://www.nature.com/articles/d41586-025-00964-w (2025)

I've always assumed its spontaneous specialization of species: leaving the safety of their nest at those times of day when they are the fittest to occupy a niche.

Once energy conserving "sleep time" exists, the genome can postpone or schedule for activity during these times of day, if it turns out to be more effective somehow.

Re: A sleep-like consolidation mechanism for LLMs

#148

Post by avadodin »

fc417fc802 wrote: post #139
Earw0rm wrote: post #115

Earlier quoted context omitted.

Probably because it evolved very early (like before bilateral symmetry, multi layer body cavity, or kidneys early... maybe even before multicellular animals early) and so has been incorporated as an essential pillar into multiple processes layered on top of that fundamental architecture.

You've merely stated observations about the context and the process that led to it. That doesn't in any way answer the question of what it's actually doing that's so essential.

He's saying that it will kill you because some other process has evolved to depend on the same genetic code.

GTAGGA turning into GTACCA may make you sleep 8 fewer hours but also keep you from producing haemoglobin.

It's like leveraging a qsort implementation from your mp3 player to develop your OS scheduling algorithm.

His argument is that there is nothing essential about its apparent function(playing mp3s).

Re: A sleep-like consolidation mechanism for LLMs

#149

Post by fc417fc802 »

avadodin wrote: post #148
fc417fc802 wrote: post #139

Earlier quoted context omitted.

You've merely stated observations about the context and the process that led to it. That doesn't in any way answer the question of what it's actually doing that's so essential.

He's saying that it will kill you because some other process has evolved to depend on the same genetic code. GTAGGA turning into GTACCA may make you sleep 8 fewer hours but also keep you from producing haemoglobin. It's like leveraging a qsort implementation from your mp3 player to develop your OS scheduling algorithm. His argument is that there is nothing essential about its apparent function(playing mp3s).

No one here said anything about genetic code.

If he articulated a particular essential process and why it depends on sleep in an incidental manner that might make for a reasonable hypothesis. However it would not refute the earlier (cited) claim that there is no consensus.

As presented without any concrete information about the processes involved it doesn't even qualify as a hypothesis, merely empty handwaving. In context it's even worse, being an entirely baseless contradiction of a claim pulled from a prominent paper.

Post reply on HN