51–60 of 69 posts
I am not trying to downplay the contribution of the paper, but isn't it obvious that this is the case?
One of Wofram's comments is that there appears to be much more internal structure in language semantics that we had expected, contra-Chomsky. We also know the brain, cortex esp, is highly recurrent, so it should be primed for creating fractals and chaotic mixing. So maybe the hidden structure is the set of neural hyperparams needed to put a given cluster of neurons into fractal/chaotic oscillations like this. Seems p…
Per some offline discussion, I'll note that while this paper is about the structure of hyperparams, it also starts with the analogy between Mandelbrot & Julia sets, Julia as the hyperparam space for Mandelbrot, the param space Well, they both also have similar fractal dimensions. Mandlebrot's Hausdorff dimension is 2, Julia is 1-2. I won't argue it here but just suggest that this is an important complexity relationsh…
It’s kind of the opposite, no? The Mandelbrot set is the set of points where the Julia set for that point includes that point.
Earlier quoted context omitted.
So what exactly are we looking at here? Did the authors only use two hyperparameters for the purpose of this visualization?
It's explained in the post: > Have you ever done a dense grid search over neural network hyperparameters? Like a really dense grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish colors to hyperparameters for which training diverges.
One of Wofram's comments is that there appears to be much more internal structure in language semantics that we had expected, contra-Chomsky. We also know the brain, cortex esp, is highly recurrent, so it should be primed for creating fractals and chaotic mixing. So maybe the hidden structure is the set of neural hyperparams needed to put a given cluster of neurons into fractal/chaotic oscillations like this. Seems p…
Text is infinitely complex. Written text is only somewhat less so. The text people choose to write is full of complex entropy.
The language patterns we use to read text, on the other hand, are much more simple. The most complicated part is ambiguity, and we don't resolve that with language. Instead, we resolve ambiguity with context.
One of Wofram's comments is that there appears to be much more internal structure in language semantics that we had expected, contra-Chomsky. We also know the brain, cortex esp, is highly recurrent, so it should be primed for creating fractals and chaotic mixing. So maybe the hidden structure is the set of neural hyperparams needed to put a given cluster of neurons into fractal/chaotic oscillations like this. Seems p…
An LLM does not model language . The name is misleading, and should be changed to Large Text Model. Text is infinitely complex. Written text is only somewhat less so. The text people choose to write is full of complex entropy. The language patterns we use to read text, on the other hand, are much more simple. The most complicated part is ambiguity, and we don't resolve that with language. Instead, we resolve ambiguit…
Earlier quoted context omitted.
An LLM does not model language . The name is misleading, and should be changed to Large Text Model. Text is infinitely complex. Written text is only somewhat less so. The text people choose to write is full of complex entropy. The language patterns we use to read text, on the other hand, are much more simple. The most complicated part is ambiguity, and we don't resolve that with language. Instead, we resolve ambiguit…
I wonder if we use LLM because LTM (and LSTM) was taken.
It's a similar problem with "AI". Artificial Intelligence does not exist, yet we put that name on all kinds of things. This causes real problems, too: by labeling an LLM "AI", we anthropomorphize it. From then on, the entire narrative is misleading.
Earlier quoted context omitted.
Per some offline discussion, I'll note that while this paper is about the structure of hyperparams, it also starts with the analogy between Mandelbrot & Julia sets, Julia as the hyperparam space for Mandelbrot, the param space Well, they both also have similar fractal dimensions. Mandlebrot's Hausdorff dimension is 2, Julia is 1-2. I won't argue it here but just suggest that this is an important complexity relationsh…
> Julia as the hyperparam space for Mandelbrot It’s kind of the opposite, no? The Mandelbrot set is the set of points where the Julia set for that point includes that point.
https://acko.net/blog/how-to-fold-a-julia-fractal/
Cheers!
Earlier quoted context omitted.
It's explained in the post: > Have you ever done a dense grid search over neural network hyperparameters? Like a really dense grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish colors to hyperparameters for which training diverges.
I saw this, but am still not clear on what the axes represent. I assume two hyperparameters, or possibly two orthogonal principal components. I guess my point is it’s not clear how/which parameters are mapped onto the image.
So what this is saying, for certain ill-chosen learning weights, model convergence is for lack of a better word, chaotic and unstable.