This is such a good explainer
Do Machine Learning Models Memorize or Generalize?
161–170 of 217 posts
Re: Do Machine Learning Models Memorize or Generalize?
#162Re: Do Machine Learning Models Memorize or Generalize?
#163Earlier quoted context omitted.
Part of the issue here is posting a LessWrong post. There is some good in there, but much of that site is like a Flat Earth conspiracy theory for neural networks. Neural network training [edit: on a fixed point task, as is often the case {such as image->label}] is always (always) biphasic necessarily, so there is no "eventual recovery from overfitting". In my experience, it is just people newer to the field or just n…
> Part of the issue here is posting a LessWrong post I mean, this whole line of analysis comes from the LessWrong community. You may disagree with them on whether AI is an existential threat, but the fact that people take that threat seriously is what gave us this whole "memorize-or-generalize" analysis, and glitch tokens before that, and RLHF before that.
I think you may be missing the extensive lines of research covering those topics. Memorization vs Generalization has been a debate before LW even existed in the public eye, and inputs that networks have unusual sensitivity to have been well studied as well (re:chaotic vs linear regimes in neural networks). Especially the memorization vs generalization bit -- that has been around for...decades. It's considered a fundamental part of the field, and has had a ton of research dedicated to it.
I don't know much either way about RLHF in terms of its direct lineage, but I highly doubt that is actually what happened, since DeepMind is actually responsible for the bulk of the historical research supporting those methods.
It's possible ala the broken clock hypothesis + LessWrong is obviously not the "primate at a typewriter" situation, so there's a chance of some people scoring meaningful contributions, but the signal to noise ratio is awful. I want to get something out of some of the posts I've tried to read there, but there are so many bad takes written with more bombastic language that it's really quite hard indeed.
Right now, it's an active detractor to the field because it pulls attention away from things that are much more deserving of energy and time. I honestly wish the vibe was back to people even just making variations of Char-RNN repos based on Karpathy's blog posts. That was a much more innocent time.
Re: Do Machine Learning Models Memorize or Generalize?
#164Sometimes I think the reason human memory in some sense is so amazing, is what we lack in storage capacity that machines have, we makeup for in our ability to create patterns that compress the amount of information stored dramatically, and then it is like we compress those patterns together with other patterns and are able to extract things from it. Like it is an incredibly lossy compression, but it gets the job done…
yes, when we do this to history, it becomes filled with conspiracies. but is merely a process to 'understand' history by projecting intentionalities. this 'compression' is what 'understanding' something really entails; at first... but then there's more. when knowledge becomes understood it enables perception (e.g. we perceive meaning in words once we learn to read). when we get really good at this understanding-perce…
This was the thinking I was taking. It is a useful tool at first but taken too far can be a bad thing in some situations.
Re: Do Machine Learning Models Memorize or Generalize?
#165Earlier quoted context omitted.
That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-bo…
I've thought about this a lot in the context of the desire people seem to have to try and achieve human immortality or at least indefinite lifespans. If SciAm is correct here and the upper bound is a quadrillion bytes, we may not be able to hit that given the bound on possible human experiences, but someone who lived long enough would eventually hit that. After a hundred million years or whatever the real number is o…
My naive assumption would be that it would be a fairly gradual process. You'd just always have a sliding window of the last N years of memories, with the older ones being progressively more fuzzy and unreliable.
Re: Do Machine Learning Models Memorize or Generalize?
#166Earlier quoted context omitted.
> but that's exactly what OP said, no? Not precisely. We don’t know if verbatim capacity is limited (and it doesn’t seem to be) but the brain operates in a space-efficient manner all the same. So there isn’t necessarily a causative relationship between “memory capacity” and “means of storage”. > Likewise, if you remembered every single word in a book, you would not understand its meaning I understand your meaning but…
> verbatim capacity is limited I am but a simple physicist and I can already tell you it is.
I mean in terms of our ability to reach those limits, naturally.
Re: Do Machine Learning Models Memorize or Generalize?
#167Earlier quoted context omitted.
That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-bo…
Does the brain require more energy to store more information? Or is it always running at the same pace regardless of if it’s empty or not? I guess the Brian doesn’t really work like that…. But I’m curious :-)
The brain doesn't seem to ever "clock down" when it's idle, which is interesting to me
https://en.wikipedia.org/wiki/Human_brain#Metabolism
> The energy consumption of the brain does not vary greatly over time
Re: Do Machine Learning Models Memorize or Generalize?
#168They digest.
Re: Do Machine Learning Models Memorize or Generalize?
#169Earlier quoted context omitted.
Does the brain require more energy to store more information? Or is it always running at the same pace regardless of if it’s empty or not? I guess the Brian doesn’t really work like that…. But I’m curious :-)
The brain doesn't seem to ever "clock down" when it's idle, which is interesting to me https://en.wikipedia.org/wiki/Human_brain#Metabolism > The energy consumption of the brain does not vary greatly over time
maybe between sleep and normal waking idle because there is actually quite a bit going on during sleep. There has been quite a bit of research though regarding higher "clock up" states consuming far more energy, such as grandmasters playing a chess tournament
Re: Do Machine Learning Models Memorize or Generalize?
#170Statistical learning can typically be phrased in terms of k nearest neighbours In the case of NNs we have a "modal knn" (memorising) going to a "mean knn" ('generalising') under the right sort of training. I'd call both of these memorising, but the latter is a kind of weighted recall. Generalisation as a property of statistical models (ie., models of conditional freqs) is not the same property as generalisation in th…
it's been proven that all models learned by gradient descent are equivalent to kernel machines. interpolation isn't generalization. if theres a new input sufficiently different from the training data the behaviour is unknown
Can you say what that says about the behavior described with the modular arithmetic in the article?
And, in particular, how to interpret the fact that different hyperparameters determined whether runs, obtaining equally high accuracy on the training data, got good or bad scores on the test data, in terms of the "view it as a kernel machine/interpolation" lens?
My understanding is that the behavior in at least one of those "models learned by gradient descent are equivalent to [some other model]" papers, works by constructing something which is based on the entire training history of the network. Is that the kernel machines one, or some other one?