academic clickbait
A sleep-like consolidation mechanism for LLMs
101–110 of 155 posts
Re: A sleep-like consolidation mechanism for LLMs
#102The "sleep" thing gives me the creeps so in my head I'm just going to think of it as the difference between "response time retrieval" and "background consolidation". I do think it points at something bigger than just attention architecture: "memory" isn't just storage, and merely longer context isn't the same thing as having a better understanding of the source data . I'm looking at this through the "personal AI" len…
I think you are missing the most important part - forgetting. The missing "memory" layers is consolidation, prioritization AND forgetting (what is not important). Also not too sure about provenance and inspectability - it is part of memory. If the source is deemed 'important' it will survive forgetting. If not, then maybe not. And its ok. I am sure you dont know the exact source who told you that the capital of Franc…
[dead]
Re: A sleep-like consolidation mechanism for LLMs
#103Earlier quoted context omitted.
> I don't think it's necessarily correct to think of sleep in terms of "it is necessary for animals or they will die". It might be more useful to think of it as "it was so useful that animals who slept outcompeted all the animals who didn't". You're talking about different things: biological necessity and evolutionary benefit. You can find out about the former by preventing an animal from sleeping (but otherwise prov…
> You can find out about the former by preventing an animal from sleeping (but otherwise provide all other needed things), and seeing if it will eventually die. That is actually almost impossible to do. The rat study was as close as we’ve ever come, and it’s still debated whether the rats died due to lack of sleep or some other mechanism, since the autopsy couldn’t confirm a cause of death. (It could have been due to…
What about fatal familial insomnia in humans
Re: A sleep-like consolidation mechanism for LLMs
#104No they do not. I'm sure that if you presented the same argument about, I don't know?, your car's CPU with built in AI; then this would be a whole different discussion entirely.
Re: A sleep-like consolidation mechanism for LLMs
#105I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities. Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?
Was it anthropomorphizing computers when they named "memory"? Seems to me like it's more analogizing for the sake of easy understanding. Sure, it's not literally the same exact mechanism, but it's certainly modeled after the biological concept.
Re: A sleep-like consolidation mechanism for LLMs
#106Re: A sleep-like consolidation mechanism for LLMs
#107sleep aka processing the data differently.
Re: A sleep-like consolidation mechanism for LLMs
#108Re: A sleep-like consolidation mechanism for LLMs
#109Isn't this simply context pruning/optimization?
From the abstract, it looks like it's actually doing something deeper, updating weights in part of the model?
The abstract and method sections only mention updating the SSM state during "sleep" (ie the same vectors that change after each token in stock Mamba) not any of the actual weight matrices. AFAICT this is just another attention compaction paper, with misleading tile? It is not very clearly written
Re: A sleep-like consolidation mechanism for LLMs
#110Earlier quoted context omitted.
> You can find out about the former by preventing an animal from sleeping (but otherwise provide all other needed things), and seeing if it will eventually die. That is actually almost impossible to do. The rat study was as close as we’ve ever come, and it’s still debated whether the rats died due to lack of sleep or some other mechanism, since the autopsy couldn’t confirm a cause of death. (It could have been due to…
What about fatal familial insomnia in humans
If I remember correctly, fatal insomnia shares most symptoms with other prion diseases (in which there might be no lack of sleep involved), so it's probably the brain damage that causes death, not insomnia itself.