The boundary of neural network trainability is fractal

Not exactly, Newton's fractal, which is topological, specifically the wada property, is a boundary condition.

It relates to fractal (non-integer) dimensions, which was first described by Mandelbrot in a paper about self similarity.

Here is a paper that covers some of that.

https://www.minvydasragulskis.com/sites/default/files/public...

In Newton's fractal, no matter how small a circle you can draw, your circle will either contain one root or all the roots.

The basins that contain one root are open sets that share a boundary set.

Even if you could have perfect information and precision this property holds. This means any change in initial conditions that crosses a boundary will be indeterminate.

There is another feature called riddled basins, where every point is arbitrarily close to other basins. This is another situation where even with perfect information and unlimited precision a perturbations would be indeterminate.

A positive Laponov exponent which isn't sufficient to prove chaos, but is always positive in the presence of chaos may even be 0 or negative in the above situations.

Take the typical predator prey model and add fear and refuge and you hit the riddled basins.

Stack four reflective balls in a pyramid and shine different color lights in two sides and you will see the Wada property.

Neither of those problems are addressable with the assumption of deterministic effects with finite precision.

> Some fractals -- for instance those associated with the Mandelbrot and quadratic Julia sets -- are computed by iterating a function, and identifying the boundary between hyperparameters for which the resulting series diverges or remains bounded. Neural network training similarly involves iterating an update function (e.g. repeated steps of gradient descent), can result in convergent or divergent behavior, and can be extremely sensitive to small changes in hyperparameters. Motivated by these similarities, we experimentally examine the boundary between neural network hyperparameters that lead to stable and divergent training. We find that this boundary is fractal over more than ten decades of scale in all tested configurations.

Reading this gave me goosebumps

I'll defend the idea that it was obvious. (Although, it wasn't obvious to me until someone pointed it out, so maybe that's not obvious.)

If you watch this video[0], you'll see in the first frame that there is a clear boundary between learning rates that converge or not. Ignoring this paper for a moment, what if we zoom in really really close to that boundary? There are two possibilities, either (1) the boundary is perfectly sharp no matter how closely we inspect it, or (2) it is a little bit fuzzy. Of those two possibilities, the perfectly sharp boundary would be more surprising.

[0]: https://x.com/jaschasd/status/1756930242965606582

So what exactly are we looking at here? Did the authors only use two hyperparameters for the purpose of this visualization?

Two quibbles: (1) neural nets don’t necessarily operate on language, (2) they only loosely model biological neurons in that they operate in discrete space. All that is to say that any similarities are purely incidental without accounting for these two facts.