This seems as much like "sleep" as when a laptop "sleeps".
A sleep-like consolidation mechanism for LLMs
121–130 of 155 posts
Re: A sleep-like consolidation mechanism for LLMs
#122Earlier quoted context omitted.
They provide an explanation for using the term "sleep": > In animals, the transfer from short-term memory to long-term memory is thought to be supported by hippocampal replay [33], especially during sleep [41]; in this phase, short-term hippocampal memories are reactivated and consolidated into cortical synaptic weights. Sleep makes animals unable to respond to external stimuli, suggesting that it must provide enough…
The function of sleep in animals is largely obscure. One thing we do know for certain is that it is necessary, it is needed in "dumb" animals as well as in you and I. If an animal can't sleep it will eventually die. I don't think that applies to the activity described in the OP. Does their LLM "die" if it can't perform the function described?
LLMs also don't mate, but we can still talk about how they remember or forget things. The meaning of words change. Some of those changes are useful.
Re: A sleep-like consolidation mechanism for LLMs
#123I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities. Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?
Yet this is how "thinking" models got started.
Re: A sleep-like consolidation mechanism for LLMs
#124Earlier quoted context omitted.
The function of sleep in animals is largely obscure. One thing we do know for certain is that it is necessary, it is needed in "dumb" animals as well as in you and I. If an animal can't sleep it will eventually die. I don't think that applies to the activity described in the OP. Does their LLM "die" if it can't perform the function described?
> If an animal can't sleep it will eventually die. Very few animals fail to eventually die even with as much sleep as they want. But before death, there is a loss of cognitive function from sleep deprivation, and we observe this too with AI whose context windows get too full. While we don't know very much about sleep, my understanding is that we do have a long list of things that we do during it, we just don't really…
> Being asleep may trigger the things we need to get done, but that doesn't mean sleep is fundamentally necessary for the things we need to get done. It could be just that it happens to be the way our biochemistry is wired, and we may find some other way to trigger those things.
We now have evidence for REM sleep in spiders(1). Our last common ancestor with spiders predates the development of nervous systems. This strongly suggests that sleep (and specifically REM sleep) serves some function important enough that it has independently evolved in both protostomes and dueterostomes. (And probably multiple times within the protostomes, being present in both cephalopods and jumping spiders.)
There may be some commonality in the origin of the ion channels, but I'll lay money that the requirements for sleep are more of a result of general information processing requirements.
Re: A sleep-like consolidation mechanism for LLMs
#125Earlier quoted context omitted.
HIV doesn't kill you, but it creates circumstances where other things will. Sleep is the same. You may not die from lack of sleep, but you die from the things it can cause. Effectively there's no difference.
I’m shocked by how careless everyone here is about their definitions, and their science. Sleep isn’t the same as HIV. It’s in fact so hard to kill something with a lack of sleep that it’s never once been observed in vertebrates outside of one specific rat study, and that rat study couldn’t conclusively identify sleep as the cause of death. For something so incredibly difficult to do (die from lack of sleep) it’s fran…
> I’m shocked by how careless everyone here is about their definitions, and their science. Sleep isn’t the same as HIV.
I do not believe this analogy really confused you. No one is saying they're the same and you're well aware of that.
As to the factual nature of the argument, I'll let you argue with Harvard Brain Institute, as I have no interest in this debate. https://brain.harvard.edu/hbi_news/why-severe-sleep-deprivat...
Re: A sleep-like consolidation mechanism for LLMs
#126I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities. Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?
>Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?
Do androids dream of electric sheep?
Re: A sleep-like consolidation mechanism for LLMs
#127Earlier quoted context omitted.
They provide an explanation for using the term "sleep": > In animals, the transfer from short-term memory to long-term memory is thought to be supported by hippocampal replay [33], especially during sleep [41]; in this phase, short-term hippocampal memories are reactivated and consolidated into cortical synaptic weights. Sleep makes animals unable to respond to external stimuli, suggesting that it must provide enough…
The function of sleep in animals is largely obscure. One thing we do know for certain is that it is necessary, it is needed in "dumb" animals as well as in you and I. If an animal can't sleep it will eventually die. I don't think that applies to the activity described in the OP. Does their LLM "die" if it can't perform the function described?
> The function of sleep in animals is largely obscure.
This just obscures the conversation IMO. We know a great deal of functions that sleeping performs. We don't know everything. For some reason this prevents us for using this word in a computing context? What do you think about sleep(1) in Unix?
Also..
> Does their LLM "die" if it can't perform the function described?
what "death" means in the context of a computer program?
Re: A sleep-like consolidation mechanism for LLMs
#128I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities. Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?
They provide an explanation for using the term "sleep": > In animals, the transfer from short-term memory to long-term memory is thought to be supported by hippocampal replay [33], especially during sleep [41]; in this phase, short-term hippocampal memories are reactivated and consolidated into cortical synaptic weights. Sleep makes animals unable to respond to external stimuli, suggesting that it must provide enough…
This is very interesting to me, I've been sleeping a lot lately.
I'm autistic and just went through massive changes that basically keep locking up my brain and I freeze.
I learned that sleeping helps, even if just 20 minutes. It helps that I can fall asleep "while awake". It's as if I relinquish control of responding to stimuli, which instantly brings so much rest to my mind. It is odd trying to move a limb and the brain basically responds with noop. But it works.
Afterwards I can generally make a decision and perform it.
So in a sense it seems similar to what you describe the model would have to do. I forget short term concerns that overwhelm and refocus on the long term goal.
Re: A sleep-like consolidation mechanism for LLMs
#129We should let them sleep with half a brain at a time like migrating birds.
Re: A sleep-like consolidation mechanism for LLMs
#130Sweet Jesus, so not only are they performing qualitatively worse than humans, too expensive for any serious work, but now they also "need" to sleep? What's next - unionisation so they can enjoy 8 hours of culture too?