11–20 of 193 posts
I'm sure there is brain activity that prepares the baby for birth, but calling it a "dream" is just clickbait
The researchers call it "dream-like" and it is, in the sense that, like a dream, the neurons are stimulated and activated as though sensory input was occurring despite there being no actual stimuli.
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.
We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.
We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
Are these innate patterns of activity priming mammalian brains in a similar way?
I always wondered how our brain is pre-wired to recognize 3D objects right after a birth, but this article is saying that eyes are giving fake image-like input to our brain so that it can perform "pre-training"? Does this imply that our brain is truly a blank slate if these initial seeds aren't given?
There was an example case of a blind man with cataracts getting it fixed at around 50 years old. People were curious if he would be able to know if a sphere was round just by looking at it without touching it.
IIRC - not only could he not do that, but he couldn't visually interpret shadows (saw them as black splotches - didn't recognize depth), and was confused why objects got smaller as they were moving away.
I think the human visual system trains on a lot of visual input data, but it typically happens at the baby stage where you can't really interact.
Earlier quoted context omitted.
The researchers call it "dream-like" and it is, in the sense that, like a dream, the neurons are stimulated and activated as though sensory input was occurring despite there being no actual stimuli.
Huh, how weird. How are those stimulis generated? Through the mother's eyes or brain activity?
I'm sure there is brain activity that prepares the baby for birth, but calling it a "dream" is just clickbait
"Mammals dream about the world they are about to experience before they are born"
Wow, those are some evocative feels in those words.
I immediately thought about Google's "DeepDream" and how reversed back propagation gives us a projection of the network. Growing brains similarly fire before they're trained on real world perceptual inputs. Maybe these have shapes that evolved to be world-like and give babies an advantage in the fitness landscape.
My next thought: what if what we're collectively doing now -- all of our neural connectomes growing, dying, forming new connections, evolving understandings. Society, basically -- is the same shape of an AGI before birth? Maybe alien worlds tend towards this path too. A bad analogy, sure, but maybe it's the shape of the universe's consciousness being bootstrapped, before it awakens. Not getting into metaphysical garbage -- just thinking about what's to come.
Freud also predicted this in that he postulated that hallucination was more primordial than actual sense-experience
Something about this reminds me of the 'Winning Ticket Hypothesis' in artifical neural networks: that some 'random' initializations prime a network far better for later faster learning. From https://arxiv.org/abs/1803.03635 - the abstract: Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of infere…
A newborn foal can stand within 55 minutes of being born and can walk or run within 90 minutes. That is crazy fast training speed. Is it training initialization or is it a form of transfer learning?
Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains that cannot mature due to birth canal constraints, which is constrained by pelvis size which is constrained by the need to walk upright.
I have wondered about how true this is. Maybe our brains are just so much more powerful, or capable of much deeper understanding, that we require a much lower initial learning rate.
The lottery ticket could explain why some children learn to walk at 6 months, while others are closer to 24 months, with no different in intelligence or motor skills in later life.
Edit: As some people below have correctly pointed out, human babies could not walk within an hour for physiological reasons, but nor do they exhibit the basic motor skills, spatial reasoning or image processing required for walking. Many human babies even struggle to feed for the first 24-48 hours or longer.
Meanwhile baboon babies can hold onto their mother from birth while the mother climbs trees.
Earlier quoted context omitted.
The researchers call it "dream-like" and it is, in the sense that, like a dream, the neurons are stimulated and activated as though sensory input was occurring despite there being no actual stimuli.
Huh, how weird. How are those stimulis generated? Through the mother's eyes or brain activity?
Dreams help the brain in generalizing the world. From the article:
"That is, dreams are a biological mechanism for increasing generalizability via the creation of corrupted sensory inputs from stochastic activity across the hierarchy of neural structures."
I always wondered how our brain is pre-wired to recognize 3D objects right after a birth, but this article is saying that eyes are giving fake image-like input to our brain so that it can perform "pre-training"? Does this imply that our brain is truly a blank slate if these initial seeds aren't given?
I don't think brains are pre-wired to recognize 3D objects. There was an example case of a blind man with cataracts getting it fixed at around 50 years old. People were curious if he would be able to know if a sphere was round just by looking at it without touching it. IIRC - not only could he not do that, but he couldn't visually interpret shadows (saw them as black splotches - didn't recognize depth), and was confu…
But I doubt it's 100% "no pre-wiring". What about gravity and musculature? Surely brains must know or train on these in some way before birth.
And what about fight or flight reflexes and presupposing monsters in the shadows? My understanding is that these had a primitive evolutionary basis and that we came hardwired to respond to certain stimuli.