I am not trying to downplay the contribution of the paper, but isn't it obvious that this is the case?
Not only it is not obvious; it is not known to be true.
The boundary of neural network trainability is fractal
41–50 of 69 posts
Re: The boundary of neural network trainability is fractal
#42Earlier quoted context omitted.
> It feels weird to me to use the hyper parameters as the variables to iterate on Yes, I also think this is strange. In regular fractals the x and y coordinates have the same units (roughly speaking), but here this is not the case, so I wonder how they determine the relative scale.
Is there really any meaningful sense in which real and imaginary numbers have the same units but two dimensionless real hyperparameters don't?
Complex numbers multiplied by an imaginary number rotate preserving magnitude. In this sense they have the same units.
Re: The boundary of neural network trainability is fractal
#43Earlier quoted context omitted.
Is there really any meaningful sense in which real and imaginary numbers have the same units but two dimensionless real hyperparameters don't?
Complex numbers multiplied by an imaginary number rotate preserving magnitude. In this sense they have the same units.
You can rotate any vector while preserving its magnitude.
Re: The boundary of neural network trainability is fractal
#44One of Wofram's comments is that there appears to be much more internal structure in language semantics that we had expected, contra-Chomsky. We also know the brain, cortex esp, is highly recurrent, so it should be primed for creating fractals and chaotic mixing. So maybe the hidden structure is the set of neural hyperparams needed to put a given cluster of neurons into fractal/chaotic oscillations like this. Seems p…
Two quibbles: (1) neural nets don’t necessarily operate on language, (2) they only loosely model biological neurons in that they operate in discrete space. All that is to say that any similarities are purely incidental without accounting for these two facts.
Well, in some sense they don’t operate on language at all - but on mathematical representation of tokens derived from language.
I’m sure from your comment you are aware of the distinction, but it is an interesting concept for people to keep in mind.
Re: The boundary of neural network trainability is fractal
#45Earlier quoted context omitted.
Complex numbers multiplied by an imaginary number rotate preserving magnitude. In this sense they have the same units.
You can rotate any vector while preserving its magnitude.
If the units are different, you do need to come up with a conversation between them, or you're implicitly saying it's 1-to-1
Re: The boundary of neural network trainability is fractal
#46One of Wofram's comments is that there appears to be much more internal structure in language semantics that we had expected, contra-Chomsky. We also know the brain, cortex esp, is highly recurrent, so it should be primed for creating fractals and chaotic mixing. So maybe the hidden structure is the set of neural hyperparams needed to put a given cluster of neurons into fractal/chaotic oscillations like this. Seems p…
Two quibbles: (1) neural nets don’t necessarily operate on language, (2) they only loosely model biological neurons in that they operate in discrete space. All that is to say that any similarities are purely incidental without accounting for these two facts.
1) agreed. it's exciting seeing the same basic architecture broadly applied
2) not sure what you mean by "operate in discrete space"
I'd emphasize the potential similarity to biological recurrence. Deep ANNs don't need to have this explicitly (tho e.g. LSTM has explicit recurrence), but it is known that recurrent NNs can be emulated by unrolling, in a process similar to function currying. In this mode, a learned network would learn to recognize certain inputs and carry them across to other parts of the network that can be copies of the originator, thus achieving functional equivalence to self feedback, or neighbor feedback. It takes a lot of layers and nodes in theory, but ofc modern nets are getting very big.
Re: The boundary of neural network trainability is fractal
#47This is much more interesting if you see the animations. https://x.com/jaschasd/status/1756930242965606582
So what exactly are we looking at here? Did the authors only use two hyperparameters for the purpose of this visualization?
It's explained in the post:
> Have you ever done a dense grid search over neural network hyperparameters? Like a really dense grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish colors to hyperparameters for which training diverges.
Re: The boundary of neural network trainability is fractal
#48Earlier quoted context omitted.
Two quibbles: (1) neural nets don’t necessarily operate on language, (2) they only loosely model biological neurons in that they operate in discrete space. All that is to say that any similarities are purely incidental without accounting for these two facts.
Well, in some sense they don’t operate on language at all - but on mathematical representation of tokens derived from language. I’m sure from your comment you are aware of the distinction, but it is an interesting concept for people to keep in mind.
[dead]
Re: The boundary of neural network trainability is fractal
#49Re: The boundary of neural network trainability is fractal
#50Even the boundary for newtons approximation is fractal. This is a feature of non-linear optimization.