What does this mean?
Training the network is a dynamic process similar to
https://en.wikipedia.org/wiki/Julia_set
or
The boundary of neural network trainability is fractal
21–30 of 69 posts
Re: The boundary of neural network trainability is fractal
#22What does this mean?
Nothing
Re: The boundary of neural network trainability is fractal
#23The blog post would be a better link for this submission. https://sohl-dickstein.github.io/2024/02/12/fractal.html
Re: The boundary of neural network trainability is fractal
#24What does this mean?
As another poster pointed out, its much more intuitive with graphics.
The parameters that you tweak to control model learning have a self-similar property where as you zoom if you see more and more complexity. Its the very definition of local maxima all over the place.
Re: The boundary of neural network trainability is fractal
#25The blog post would be a better link for this submission. https://sohl-dickstein.github.io/2024/02/12/fractal.html
It was posted on HN last week: https://news.ycombinator.com/item?id=39349992
Re: The boundary of neural network trainability is fractal
#26I am not trying to downplay the contribution of the paper, but isn't it obvious that this is the case?
Obvious to whom?
Re: The boundary of neural network trainability is fractal
#27I vaguely recall that in vivo neural oscillations also exhibit fractal structure (in some cases at least).
Re: The boundary of neural network trainability is fractal
#28Reposting comment from last time since I'm still curious: This is really fun to see. I love toy experiments like this. I see that each plot is always using the same initialization of weights, which presumably makes it possible to have more smoothness between each pixel. I also would guess it's using the same random seed for training (shuffling data). I'd be curious to know what the plots would look like with a differ…
I think if you used a random seed for weights and training data order, and reran the experiment enough times to average out the noise, then the resulting charts would then be smooth with no fractal patterns.
An interesting conjecture, well worth a paper in response.
Re: The boundary of neural network trainability is fractal
#29One of Wofram's comments is that there appears to be much more internal structure in language semantics that we had expected, contra-Chomsky.
We also know the brain, cortex esp, is highly recurrent, so it should be primed for creating fractals and chaotic mixing.
So maybe the hidden structure is the set of neural hyperparams needed to put a given cluster of neurons into fractal/chaotic oscillations like this. Seems potentially more useful too.. way more information content than a configuration that yields a fast convergence to a fixed point.
Perhaps this is what learning deep NNs is doing: producing conditions where the substrate is at the tipping point, to get to a high-information generation condition, and then shaping this to fit the target system as well as it can with so many free parameters.
That suggests that using iterative generators that are somehow closer to the dynamics of real neurons would be more efficient for AI: it'd be easier to drive them to similar feedback conditions and patterns
Like matching resonators in any physical system
Re: The boundary of neural network trainability is fractal
#30This is much more interesting if you see the animations. https://x.com/jaschasd/status/1756930242965606582
Fractal zoom videos are worth infinite words.