Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A sleep-like consolidation mechanism for LLMs

arxiv.org

Post reply on HN
Print view View original

61–70 of 155 posts

Re: A sleep-like consolidation mechanism for LLMs

#61

Post by bmc7505 »

This topic recently came up at the FLANN workshop [1], and seems to periodically be rediscovered [2,3,4] in different contexts. While some have speculated about the biological role it plays (e.g., Pearlmutter & Houghton [5]), we still lack a conclusive theory of sleep, but the convergent evolution of this specific phenomenon across the animal kingdom and the fact that deprivation is inevitably fatal seems like an important clue.

[1]: https://flann.cs.yale.edu

[2]: https://www.cs.toronto.edu/~hinton/csc2535/readings/ws.pdf

[3]: https://arxiv.org/abs/1711.02282

[4]: https://arxiv.org/abs/2006.08381

[5]: https://mural.maynoothuniversity.ie/id/eprint/1653/1/Hamilto...

Re: A sleep-like consolidation mechanism for LLMs

#62

Post by dijit »

sillysaurusx wrote: post #59
dijit wrote: post #56

Earlier quoted context omitted.

> No human has ever died from lack of sleep. As far as I understand it, there is a disease that destroys your brain's ability to produce sleep. Once you have it, you suffer total, progressive insomnia and die within roughly 6–18 months. Scientists debate whether it's the underlying brain damage or the sleeplessness itself that causes death, but the two are inseparable in practice, and sleep deprivation is considered…

My second paragraph addresses that: > People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep. It’s a prion disease. It’s established fact that they don’t die from the lack of sleep.

Interesting that the scientific debate is settled, because you said so. Researchers who study prion diseases would probably be surprised to hear it.

Re: A sleep-like consolidation mechanism for LLMs

#64

Post by refulgentis »

sillysaurusx wrote: post #43
pcrh wrote: post #20

Earlier quoted context omitted.

The function of sleep in animals is largely obscure. One thing we do know for certain is that it is necessary, it is needed in "dumb" animals as well as in you and I. If an animal can't sleep it will eventually die. I don't think that applies to the activity described in the OP. Does their LLM "die" if it can't perform the function described?

> If an animal can't sleep it will eventually die. That turns out to be un-settled science. No human has ever died from lack of sleep. People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep. In a series of controlled experiments, rats and fruit flies did die from lack of sleep. But no one has yet proven that it holds true for vertebrates except for rats. In o…

It’s a bunch of Claude blather, and I love Claude. Just not worth copying over to HN, because the rush to get to a narrow answer to a narrow question elides the meaningful bits, ex. what does happen during sleep deprivation. Has a “not even wrong” air simply because you’re trying to get to true/false on a narrow question then pushing your research assistant to disavow what you’re quote unquote “skeptical” of.

Re: A sleep-like consolidation mechanism for LLMs

#65

Post by sillysaurusx »

burnte wrote: post #57
sillysaurusx wrote: post #43

Earlier quoted context omitted.

> If an animal can't sleep it will eventually die. That turns out to be un-settled science. No human has ever died from lack of sleep. People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep. In a series of controlled experiments, rats and fruit flies did die from lack of sleep. But no one has yet proven that it holds true for vertebrates except for rats. In o…

HIV doesn't kill you, but it creates circumstances where other things will. Sleep is the same. You may not die from lack of sleep, but you die from the things it can cause. Effectively there's no difference.

I’m shocked by how careless everyone here is about their definitions, and their science. Sleep isn’t the same as HIV. It’s in fact so hard to kill something with a lack of sleep that it’s never once been observed in vertebrates outside of one specific rat study, and that rat study couldn’t conclusively identify sleep as the cause of death.

For something so incredibly difficult to do (die from lack of sleep) it’s frankly crazy that most people here are saying it like it’s fact.

Re: A sleep-like consolidation mechanism for LLMs

#66

Post by incognito124 »

sillysaurusx wrote: post #50
DonHopkins wrote: post #31

Earlier quoted context omitted.

Is it "Anthropicmorphization" when Claud treats human beings like LLMs?

Interesting question. Is there an actual term for that? It’s like inverse anthropomorphization, but not quite.

Mechanomorphisation

Re: A sleep-like consolidation mechanism for LLMs

#67

Post by sillysaurusx »

dijit wrote: post #62
sillysaurusx wrote: post #59

Earlier quoted context omitted.

My second paragraph addresses that: > People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep. It’s a prion disease. It’s established fact that they don’t die from the lack of sleep.

Interesting that the scientific debate is settled, because you said so. Researchers who study prion diseases would probably be surprised to hear it.

Huh? Ask Claude or do some research on the topic if you don’t believe me. A prion disease killing you has nothing whatsoever to do with the lack of sleep. The insomnia is a side effect, not the cause.

Jeez. People here are really stretching to defend their false “we die without sleep” claim.

Re: A sleep-like consolidation mechanism for LLMs

#68

Post by sillysaurusx »

refulgentis wrote: post #64
sillysaurusx wrote: post #43

Earlier quoted context omitted.

> If an animal can't sleep it will eventually die. That turns out to be un-settled science. No human has ever died from lack of sleep. People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep. In a series of controlled experiments, rats and fruit flies did die from lack of sleep. But no one has yet proven that it holds true for vertebrates except for rats. In o…

It’s a bunch of Claude blather, and I love Claude. Just not worth copying over to HN, because the rush to get to a narrow answer to a narrow question elides the meaningful bits, ex. what does happen during sleep deprivation. Has a “not even wrong” air simply because you’re trying to get to true/false on a narrow question then pushing your research assistant to disavow what you’re quote unquote “skeptical” of.

This is little more than a fancy way of saying “Nu uh.” Such arguments are hardly convincing.

Re: A sleep-like consolidation mechanism for LLMs

#69

Post by madibo3156 »

pcrh wrote: post #4

I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities. Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?

I find this annoying too. "Sleep" is okay, but the quippy headlines ("need sleep"—short, snappy and vague) infiltrating journals bother me. I've seen it well before LLMs, but as an example, there is a long list of title snowclones of the famous attention paper: https://github.com/vinayprabhu/X-is-all-you-need.

Re: A sleep-like consolidation mechanism for LLMs

#70

Post by nkmnz »

sillysaurusx wrote: post #65
burnte wrote: post #57

Earlier quoted context omitted.

HIV doesn't kill you, but it creates circumstances where other things will. Sleep is the same. You may not die from lack of sleep, but you die from the things it can cause. Effectively there's no difference.

I’m shocked by how careless everyone here is about their definitions, and their science. Sleep isn’t the same as HIV. It’s in fact so hard to kill something with a lack of sleep that it’s never once been observed in vertebrates outside of one specific rat study, and that rat study couldn’t conclusively identify sleep as the cause of death. For something so incredibly difficult to do (die from lack of sleep) it’s fran…

I'd probably kill myself after a couple of days without sleep. Would the lack of sleep be the cause of death or the cause of the cause of death?
Post reply on HN