The boundary of neural network trainability is fractal
1–10 of 69 posts
Re: The boundary of neural network trainability is fractal
#2[flagged]
Re: The boundary of neural network trainability is fractal
#3Reposting comment from last time since I'm still curious:
This is really fun to see. I love toy experiments like this. I see that each plot is always using the same initialization of weights, which presumably makes it possible to have more smoothness between each pixel. I also would guess it's using the same random seed for training (shuffling data). I'd be curious to know what the plots would look like with a different randomness/shuffling of each pixel's dataset. I'd guess for the high learning rates it would be too noisy, but you might see fractal behavior at more typical and practical learning rates. You could also do the same with the random initialization of each dataset. This would get at if the chaotic boundary also exists in more practical use cases.
Re: The boundary of neural network trainability is fractal
#4It feels weird to me to use the hyper parameters as the variables to iterate on, and also wasteful. Surely there must be a family of models that give fractal like behaviour ?
Re: The boundary of neural network trainability is fractal
#5Related: https://news.ycombinator.com/item?id=39349992 (seems to be the same content)
Re: The boundary of neural network trainability is fractal
#6Could it be that this behavior is just caused by numerical issues and/or randomness in the calculation, rather than a real property of neural networks?
Re: The boundary of neural network trainability is fractal
#7It feels weird to me to use the hyper parameters as the variables to iterate on, and also wasteful. Surely there must be a family of models that give fractal like behaviour ?
> It feels weird to me to use the hyper parameters as the variables to iterate on
Yes, I also think this is strange. In regular fractals the x and y coordinates have the same units (roughly speaking), but here this is not the case, so I wonder how they determine the relative scale.
Re: The boundary of neural network trainability is fractal
#8In case it isn't obvious, you can tap on any of the figures in the PDF or HTML version to watch the video.
Re: The boundary of neural network trainability is fractal
#9Reposting comment from last time since I'm still curious: This is really fun to see. I love toy experiments like this. I see that each plot is always using the same initialization of weights, which presumably makes it possible to have more smoothness between each pixel. I also would guess it's using the same random seed for training (shuffling data). I'd be curious to know what the plots would look like with a differ…
exactly. if they always use the same initialization seed then this isn't very surprising.
one would have to do many runs for each point in the grid and average them or something.
but i didn't read the paper so maybe they did.
Re: The boundary of neural network trainability is fractal
#10Reposting comment from last time since I'm still curious: This is really fun to see. I love toy experiments like this. I see that each plot is always using the same initialization of weights, which presumably makes it possible to have more smoothness between each pixel. I also would guess it's using the same random seed for training (shuffling data). I'd be curious to know what the plots would look like with a differ…
I think if you used a random seed for weights and training data order, and reran the experiment enough times to average out the noise, then the resulting charts would then be smooth with no fractal patterns.