41–50 of 217 posts
Sometimes I think the reason human memory in some sense is so amazing, is what we lack in storage capacity that machines have, we makeup for in our ability to create patterns that compress the amount of information stored dramatically, and then it is like we compress those patterns together with other patterns and are able to extract things from it. Like it is an incredibly lossy compression, but it gets the job done…
Statistical learning can typically be phrased in terms of k nearest neighbours In the case of NNs we have a "modal knn" (memorising) going to a "mean knn" ('generalising') under the right sort of training. I'd call both of these memorising, but the latter is a kind of weighted recall. Generalisation as a property of statistical models (ie., models of conditional freqs) is not the same property as generalisation in th…
We have suspected that neural nets are a kind of kNN. Here's a paper:
Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
https://en.wikipedia.org/wiki/Percolation_theory
A relevant, recent paper I found from a quick search: The semantic landscape paradigm for neural networks (https://arxiv.org/abs/2307.09550)
Statistical learning can typically be phrased in terms of k nearest neighbours In the case of NNs we have a "modal knn" (memorising) going to a "mean knn" ('generalising') under the right sort of training. I'd call both of these memorising, but the latter is a kind of weighted recall. Generalisation as a property of statistical models (ie., models of conditional freqs) is not the same property as generalisation in th…
it's been proven that all models learned by gradient descent are equivalent to kernel machines. interpolation isn't generalization. if theres a new input sufficiently different from the training data the behaviour is unknown
Sometimes I think the reason human memory in some sense is so amazing, is what we lack in storage capacity that machines have, we makeup for in our ability to create patterns that compress the amount of information stored dramatically, and then it is like we compress those patterns together with other patterns and are able to extract things from it. Like it is an incredibly lossy compression, but it gets the job done…
[0]: https://www.scientificamerican.com/article/new-estimate-boos...
Earlier quoted context omitted.
I have observed this pattern before in computer vision tasks (train accuracy flatlining for a while before test acc starts to go up). The point of the simple tasks is to be able to interpret what could be going on behind the scenes when this happens.
No doubt. But I have also seen what people thought were generalized models failing on outlier, but valid, data. Quite often. Put another way, it isn't just how simple this task seems to be in the number of terms that are important, but isn't it also a rather dense function? Probably better question to ask is how sensitive are models that are looking at less dense functions to this? (Or more dense.). I'm not trying to…
hierarchize would be a better term than generalize
Anything would be better than "grokking". From what I gather they're talking about double descent which afaik is the consequence of overparameterization leading to a smooth interpolation between the training data as opposed to what happens in traditional overfitting. Imagine a polynomial fit with the same degree as the number of data points (swinging up and down wildly away from the data) compared with a much higher…
Scientists are also pretty lousy at making new discoveries without labs. They just need training data.
If your data set is too small, they memorize. If you train them well on a large dataset, they learn to generalize.
It generalized splendidly - it's conclusion was that you always need to press "forward" and do nothing else, no matter what happens :)