Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The boundary of neural network trainability is fractal

arxiv.org

Post reply on HN
Print view View original

11–20 of 69 posts

Re: The boundary of neural network trainability is fractal

#11

Post by JL-Akrasia »

I'll add to this.

It's not only the boundary that is fractal.

We'll soon see that learning on one dataset (area of fractal) with enough data will generalize to other seemingly unrelated datasets.

There is evidence that the structure neural networks are learning to approximate in a generative fractal of sorts.

Finally, we'll need to adapt gradient descent to operate at move between different scales

Re: The boundary of neural network trainability is fractal

#13

Post by freeone3000 »

jerpint wrote: post #4

It feels weird to me to use the hyper parameters as the variables to iterate on, and also wasteful. Surely there must be a family of models that give fractal like behaviour ?

The fractal behaviour is an undesirable property, not the goal :P ideally every network would be trainable (would converge)! this is the graphed result of a hyperparameter search, a form of optimization in neural networks.

If you envision a given architecture as a class of (higher-order) function, the inputs would be the parameters, and the constants would be the hyperparameters. Varying the constants moves to a different function in the class, or, varying the hyperparameters gives a different model with the same architecture (even with the same data).

Re: The boundary of neural network trainability is fractal

#14

Post by sigmoid10 »

blackbear_ wrote: post #6

Could it be that this behavior is just caused by numerical issues and/or randomness in the calculation, rather than a real property of neural networks?

Fractals are not caused by randomness. Fractals arise from scale invariance and self similarity, which in turn can come from non linear systems and iterative processes. It is very easy to generate fractals in practice and even the most vanilla neural networks trivially fulfill the requirements (at least when you look at training output). In that sense it would be weird not to find fractal structures when you look hard enough.

Re: The boundary of neural network trainability is fractal

#15

Post by thfuran »

amelius wrote: post #7
jerpint wrote: post #4

It feels weird to me to use the hyper parameters as the variables to iterate on, and also wasteful. Surely there must be a family of models that give fractal like behaviour ?

> It feels weird to me to use the hyper parameters as the variables to iterate on Yes, I also think this is strange. In regular fractals the x and y coordinates have the same units (roughly speaking), but here this is not the case, so I wonder how they determine the relative scale.

Is there really any meaningful sense in which real and imaginary numbers have the same units but two dimensionless real hyperparameters don't?

Re: The boundary of neural network trainability is fractal

#18

Post by tomxor »

sigmoid10 wrote: post #14
blackbear_ wrote: post #6

Could it be that this behavior is just caused by numerical issues and/or randomness in the calculation, rather than a real property of neural networks?

Fractals are not caused by randomness. Fractals arise from scale invariance and self similarity, which in turn can come from non linear systems and iterative processes. It is very easy to generate fractals in practice and even the most vanilla neural networks trivially fulfill the requirements (at least when you look at training output). In that sense it would be weird not to find fractal structures when you look har…

"If we're built from spirals, while living in a giant spiral, then everything we put our hands to, is infused with the spiral"

... Sorry I couldn't help myself.

Post reply on HN