Live data from Hacker News

A sleep-like consolidation mechanism for LLMs

arxiv.org

111–120 of 155 posts

Re: A sleep-like consolidation mechanism for LLMs

#111
post #56

Earlier quoted context omitted.

> If an animal can't sleep it will eventually die. That turns out to be un-settled science. No human has ever died from lack of sleep. People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep. In a series of controlled experiments, rats and fruit flies did die from lack of sleep. But no one has yet proven that it holds true for vertebrates except for rats. In o…

> No human has ever died from lack of sleep. As far as I understand it, there is a disease that destroys your brain's ability to produce sleep. Once you have it, you suffer total, progressive insomnia and die within roughly 6–18 months. Scientists debate whether it's the underlying brain damage or the sleeplessness itself that causes death, but the two are inseparable in practice, and sleep deprivation is considered…

Fatal Familial Insomnia is an incredibly rare prion disease that causes widespread neurological destruction. It's not remotely a normal brain that has chosen not to sleep. It's such a highly non-trivial deviation of the brain that we've only identified a few dozen families in the entire planet that suffer from it. At this point, quite a lot of things have already gone wrong in your brain.

There is quite literally no prion disease that isn't fatal.

Sleep does a lot of very important things that we probably wouldn't live long without, but it really is unclear to what extent sleep is necessary for them. If we had enough knowledge, could we trigger all the things sleep does without invoking sleep itself ? Perhaps sleep is just a very convenient mechanism.

Re: A sleep-like consolidation mechanism for LLMs

#112
post #4

I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities. Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?

First, this is not a "debate over the abilities" of LLMs. It's a proposed method to improve their performance, and the authors are free to call it however they think it makes sense.

Second, explicitly avoiding things that sound like anthropomorphisation is equally not helpful- why avoid a metaphor that works?

Third, it's really a pity that this pointless nitpicking is dominating the thread.

Re: A sleep-like consolidation mechanism for LLMs

#113

The idea of periodically stopping to write blocks of recent context into a fast-weight state is interesting, but I think it liked it better when E2E-TTT[1] did it. It's a more flexible and elegant continuous learning approach. Essentially it goes "You know how your model can remember its training data? Well, what if you treated its recent context like more training data and updated (some of) the weights using (mostly…

Yah I think E2E-TTT is a lot more like what people in this comments section are picturing. I can't tell that this method updates model weights at all during the "sleep" period, only the usual SSM state updated by any Mamba model after each token. They just optimized the model to use that SSM state _more_ when an eviction is about to happen.

Re: A sleep-like consolidation mechanism for LLMs

#115
post #90

Earlier quoted context omitted.

There is a lot that is known about sleep. We don't know everything and there are large gaps in our knowledge, but there is also a lot that we do know. And this research explicitly tried to emulate the things we know that sleep does do. Calling it "sleep" is warranted, imho.

"Despite myriad studies, there is still no consensus on why sleep is needed for survival." https://www.nature.com/articles/d41586-025-00964-w (2025)

Probably because it evolved very early (like before bilateral symmetry, multi layer body cavity, or kidneys early... maybe even before multicellular animals early) and so has been incorporated as an essential pillar into multiple processes layered on top of that fundamental architecture.

Re: A sleep-like consolidation mechanism for LLMs

#116
post #4

I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities. Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?

That's because the purpose of this article is not to have an objective debate over their abilities at all. Most interesting research in this field isn't. Instead, it's to present a new technique to improve LLM performance, which is much more interesting than (once again) rehashing the philosophy of LLM personhood.

Re: A sleep-like consolidation mechanism for LLMs

#118
post #4

I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities. Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?

When the goal of that function is to think (a notoriously human behavior), it's perfectly understandable to anthropomorphize it.

Re: A sleep-like consolidation mechanism for LLMs

#119
post #4

I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities. Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?

Please re-read up to the end of page 2 and then re-ask this question.

Re: A sleep-like consolidation mechanism for LLMs

#120

The entire industry is so desperate to anthropomorphize. What the paper describes is an offline recurrent consolidation phase: the model runs multiple forward passes over recently accumulated context, updates persistent fast weights in SSM blocks, then clears the KV cache before continuing. It has absolutely nothing to do with sleeping, but I believe the authors had a goal in mind when creating this title, and it was…

It is a descriptive analogy, get over yourself.
Post reply on HN