Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do Machine Learning Models Memorize or Generalize?

pair.withgoogle.com

Post reply on HN
Print view View original

191–200 of 217 posts

Re: Do Machine Learning Models Memorize or Generalize?

#191

Post by tbalsam »

ShamelessC wrote: post #118
tbalsam wrote: post #109

Earlier quoted context omitted.

Part of the issue here is posting a LessWrong post. There is some good in there, but much of that site is like a Flat Earth conspiracy theory for neural networks. Neural network training [edit: on a fixed point task, as is often the case {such as image->label}] is always (always) biphasic necessarily, so there is no "eventual recovery from overfitting". In my experience, it is just people newer to the field or just n…

> Part of the issue here is posting a LessWrong post. There is some good in there, but much of that site is like a Flat Earth conspiracy theory for neural networks. Indeed! It’s very frustrating that so many people here are such staunch defenders of LessWrong. Some/much of the behavior there is honestly concerning.

100% agreed. I'm pretty sure today was the first time I learned that the site was founded by Yudkowsky, which honestly explains quite a bit (polite 'lol' added here for lightheartedness)

Re: Do Machine Learning Models Memorize or Generalize?

#192

Post by NikkiA »

snewman wrote: post #65
gorjusborg wrote: post #35

Earlier quoted context omitted.

Yeah, there is definitely irony that I'm trying to push my own definition of an extra-terrestrial word, complaining that someone is ruining it. If anyone wants to come up with their own definition, read Robert Heinlein's 'Stranger in a Strange Land'. There is no definition in there, but you build an intuition of the meaning by its use. One of the issues I have w/ the use in AI is that using the word 'grok' suggests t…

Actually the definition of 'grok' is discussed in the book; you can find some relevant snippets at https://en.m.wikipedia.org/wiki/Grok . My recollection is that the book says the original / literal meaning is "drink", but this isn't supported by the Wikipedia quotes and perhaps I am misremembering, it has been a long time.

The book also points out that it is much more than just 'drink', and 'drink' would by no means cover 99% of the way it is used in the book.

That said, I've only ever read the full unabridged re-issue from the mid-90s, it's possible the earlier, edited, releases had many of the uses elided.

Re: Do Machine Learning Models Memorize or Generalize?

#193

Post by tysam_and »

Jack000 wrote: post #7
version_five wrote: post #3

Earlier quoted context omitted.

Anything would be better than "grokking". From what I gather they're talking about double descent which afaik is the consequence of overparameterization leading to a smooth interpolation between the training data as opposed to what happens in traditional overfitting. Imagine a polynomial fit with the same degree as the number of data points (swinging up and down wildly away from the data) compared with a much higher…

double descent is a different phenomenon from grokking

Nope, they are the same, just that grokking is when the KL between the representable information of the implicit biases and the data is extremely high (i.e. the network is poorly-designed or oriented for the task).

It's an informal term that not everyone accepts. Double-descent is acceptable as it describes a general phenomenon that is a natural consequence of a phase transition during neural network training. Grokking is like, to me, the 'fetch' of neural network terms. It's not new, it adds a seeming layer of separation from double-descent (which is is -- just very delayed), and it's not really accepted by everyone.

I personally do not like it at all. Especially because language affects _our_ implicit biases about what neural networks can and cannot do. We've already seen that their capacities and performance can be pushed way beyond what we traditionally expect of them.

But to summarize, they are the same. And this is why we need good terminology, as well, because poor adoption and boosting of improper terminology induces excess regret in the information exchange surface between agents in a game-theoretic sense in this lovely landscape of the ML world.

Re: Do Machine Learning Models Memorize or Generalize?

#194

Post by woopwoop »

PoignardAzur wrote: post #154
tbalsam wrote: post #109

Earlier quoted context omitted.

Part of the issue here is posting a LessWrong post. There is some good in there, but much of that site is like a Flat Earth conspiracy theory for neural networks. Neural network training [edit: on a fixed point task, as is often the case {such as image->label}] is always (always) biphasic necessarily, so there is no "eventual recovery from overfitting". In my experience, it is just people newer to the field or just n…

> Part of the issue here is posting a LessWrong post I mean, this whole line of analysis comes from the LessWrong community. You may disagree with them on whether AI is an existential threat, but the fact that people take that threat seriously is what gave us this whole "memorize-or-generalize" analysis, and glitch tokens before that, and RLHF before that.

I don't think that is true? As far as I know the grokking phenomenon was first observed (and the name coined) in this paper, not in any blog post:

https://arxiv.org/abs/2201.02177

Re: Do Machine Learning Models Memorize or Generalize?

#195

Post by ben_w »

kaba0 wrote: post #184
riversflow wrote: post #91

Earlier quoted context omitted.

I see this brought up consistently on the topic of AI take-off/X-risk. How does an AI language model devise an experiment and observe the results? The language model is only trained on what’s already known, I’m extremely incredulous that this language model technique can actually reason a genuinely novel hypothesis. A LLM is a series of weights sitting in the ram of GPU cluster, it’s really just a fancy prediction fu…

> Do we even have a model or even an idea about what “thinking” is At the least, it is a computable function (as we don’t have any physical system that would be more general than that, though some religions might disagree). Which already puts human brains ahead of LLM systems, as we are Turing-complete, while LLMs are not, at least in their naive application (their output can be feeded to subsequent invocations and t…

Is it provably not turning complete? That property pops up everywhere even when not intended, like Magic: The Gathering card interactions.

Technically you may not want to call it Turing complete given the limited context window, but I'd say that's like insisting a Commodore 64 isn't Turing complete for the same reason.

Likewise the default settings may be a bit too random to be a Turing machine, but that criticism would also apply to a human.

Re: Do Machine Learning Models Memorize or Generalize?

#196

Post by 3cats-in-a-coat »

Chabsff wrote: post #78
3cats-in-a-coat wrote: post #72

Earlier quoted context omitted.

Kinda fuzzy what's "in the distribution", because it depends on how deeply the model interprets it. If it understands examples outside the distribution... that kinda puts them in the distribution. General understanding makes the information in the distribution very wide. Shallow understanding makes it very narrow. Like say recognizing only specific combinations of pixels verbatim.

I think you are misinterpreting. The distribution present in the training set in isolation (the one I'm referring to, and is not fuzzy in the slightest) is not the same thing as the distribution understood by the trained model (the one you are referring to, and is definitely more conceptual and hard to characterize in non-trivial cases). "Generalization" is simply the theoretical measure of how much the later extends…

I'm saying how you determine the distribution in the training set depends on what the model understands and what the people who selected the dataset understand.

There's no distribution of meaning in the training set that's independent of interpretation and understanding. Aside from maybe the literal series of bits (and words and pixels) in it, as encoded.

In statistics that is not as severe a problem because you can plot how the data distribution lies in a commonly agreed upon position in one or more clearly defined and agreed upon dimensions. And you can look at the chart and talk about this shared interpretation objectively, and its distribution.

Although as a matter of fact just as often it matters what questions you asked, and how and when and whom you asked, for the distribution of answers you got. Lying with statistics is easy as it's full of hidden variables. This is why statistics is great when the data is simple and the analysis is simple, mathematical, objective, but social studies tend to yield, whatever you want them to yield.

So. What dimensions are we talking about with a self-evolved model? You have some understanding of what the data is, subjective to you. Maybe your team has some shared understanding of what the data covers, you have overlap. But the model has its own understanding, evolved independently. How much does it overlap with you? Not as much as you think.

It's a problem decades old, that people give to the model data that contains things they didn't realize it contains. They themselves didn't see that. And then get surprised by the results.

Say when an apple falls on your head, did you realize this contains the data required to describe classic mechanics? For centuries, billions of people didn't realize. To Newton it was there as clear as daylight. In the apple's fall. I know, the example is a myth, but the principle stands.

Another example, a video of the change of light patterns reflected on the floor around the corner of room where a person, out of frame, is writing on a computer. What does this data contain? You think nothing much. Maybe it contains how a floor looks. To a model, it can easily also contain what the person who is not in frame, wrote on their keyboard.

So given all this... what IS in the distribution? Depends with whose eyes you're looking. Your eyes are not the most objective eyes, nor the most intelligent eyes. You have no anchor to point to as the ultimate arbiter of what complex data contains or does not.

Re: Do Machine Learning Models Memorize or Generalize?

#197

Post by jjk166 »

staunton wrote: post #156
jjk166 wrote: post #133

Earlier quoted context omitted.

Memorization is storing data. Generalization is developing the heuristics by which you compress stored data. To distill knowledge is to apply heuristics to lossily-compress a large amount of data to a much smaller amount of data from which you nevertheless can recover enough information to be useful in the future.

> Generalization is developing the heuristics by which you compress stored data such that > you nevertheless can recover enough information to be useful in the future. I disagree (in case you meant to imply it) that compression implies generalization.

I did not mean to imply compression implies generalization, if anything the reverse. Compression is the act of cutting, generalization is the whetstone by which you may sharpen a blade, which is the compression heuristic. A more general heuristic is to compression what a sharper blade is to cutting.

Re: Do Machine Learning Models Memorize or Generalize?

#198

Post by swid »

kaba0 wrote: post #184
riversflow wrote: post #91

Earlier quoted context omitted.

I see this brought up consistently on the topic of AI take-off/X-risk. How does an AI language model devise an experiment and observe the results? The language model is only trained on what’s already known, I’m extremely incredulous that this language model technique can actually reason a genuinely novel hypothesis. A LLM is a series of weights sitting in the ram of GPU cluster, it’s really just a fancy prediction fu…

> Do we even have a model or even an idea about what “thinking” is At the least, it is a computable function (as we don’t have any physical system that would be more general than that, though some religions might disagree). Which already puts human brains ahead of LLM systems, as we are Turing-complete, while LLMs are not, at least in their naive application (their output can be feeded to subsequent invocations and t…

I googled whether or not universal function approximators, which neural nets are considered, are also considered Turing complete. It seems the general consensus is kind of not, since they are continuous and can’t do discreet operations in the same way.

But also, that isn’t quite the whole story, since they can be arbitrarily precise in their approximation. Here[0] is a white paper addressing this issue which concludes attention networks are Turing complete.

0: https://jmlr.org/papers/volume22/20-302/20-302.pdf

Re: Do Machine Learning Models Memorize or Generalize?

#200

Post by tbalsam »

woopwoop wrote: post #194
PoignardAzur wrote: post #154

Earlier quoted context omitted.

> Part of the issue here is posting a LessWrong post I mean, this whole line of analysis comes from the LessWrong community. You may disagree with them on whether AI is an existential threat, but the fact that people take that threat seriously is what gave us this whole "memorize-or-generalize" analysis, and glitch tokens before that, and RLHF before that.

I don't think that is true? As far as I know the grokking phenomenon was first observed (and the name coined) in this paper, not in any blog post: https://arxiv.org/abs/2201.02177

That's true, and I probably should have done some better backing up, sorting out, and clarification. I remember when that paper came out, it rubbed me the wrong way too then, because it is people rediscovering double descent from a different perspective, and not recognizing it as such.

What it would be better defined as is "a sudden change in phase state after a long period of metastability". Even then it ignores that those sharp inflections indicate a poor KL between some of the inductive priors and the data at hand.

You can think about it as the loss signal from the support of two gaussians extremely far apart with narrow standard deviations. Sure, they technically have support, but in a noisy regime you're going to have nothing.... nothing.... nothing....and then suddenly something as you hit that point of support.

Little of the literature, definitions around the word, or anything like that really takes this into account generally, leading to this mass illusion that this is not a double descent phenomenon, when in fact it is.

Hopefully this is a more appropriate elaboration, I appreciate your comment pointing out my mistake.

Post reply on HN