Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do Machine Learning Models Memorize or Generalize?

pair.withgoogle.com

Post reply on HN
Print view View original

131–140 of 217 posts

Re: Do Machine Learning Models Memorize or Generalize?

#131

Post by TheRealSteel »

ComputerGuru wrote: post #45
mostertoaster wrote: post #37

Sometimes I think the reason human memory in some sense is so amazing, is what we lack in storage capacity that machines have, we makeup for in our ability to create patterns that compress the amount of information stored dramatically, and then it is like we compress those patterns together with other patterns and are able to extract things from it. Like it is an incredibly lossy compression, but it gets the job done…

That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-bo…

You seem to have just re-stated what the other person said.

Re: Do Machine Learning Models Memorize or Generalize?

#132

Post by TheRealSteel »

svachalek wrote: post #101
bufferoverflow wrote: post #79

Earlier quoted context omitted.

There are rare people who remember everything https://youtu.be/hpTCZ-hO6iI

It's pretty fascinating to me how "normal" Marilu Henner seems to be. I'm getting older and my memory is not what it was, but when I was younger it was pretty extraordinary. I did really well in school and college but over time I've realized it was mostly due to being able to remember most things pretty effortlessly, over being truly "smart" in a classic sense. But having so much of the past being so accessible is to…

" I did really well in school and college but over time I've realized it was mostly due to being able to remember most things pretty effortlessly"

Same! They thought I was a genius in primary school but I ended up a loser adult with a dead end job. Turns out I just liked technology and was good at remembering facts and names for things.

Re: Do Machine Learning Models Memorize or Generalize?

#133

Post by jjk166 »

w10-1 wrote: post #98
jjk166 wrote: post #74

Earlier quoted context omitted.

Distilling knowledge is data compression.

You're conflating memorization with generalization, no?

Memorization is storing data. Generalization is developing the heuristics by which you compress stored data. To distill knowledge is to apply heuristics to lossily-compress a large amount of data to a much smaller amount of data from which you nevertheless can recover enough information to be useful in the future.

Re: Do Machine Learning Models Memorize or Generalize?

#134

Post by ComputerGuru »

halflings wrote: post #82
ComputerGuru wrote: post #45

Earlier quoted context omitted.

That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-bo…

> That’s not exactly true [...] Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components ...but that's exactly what OP said, no? I remember attending an ML presentation where the speaker shared a quote I can't find anymore (speaking of memory and generalization :)), which said something like: "To learn is to forget" If we memorized everyt…

> but that's exactly what OP said, no?

Not precisely. We don’t know if verbatim capacity is limited (and it doesn’t seem to be) but the brain operates in a space-efficient manner all the same. So there isn’t necessarily a causative relationship between “memory capacity” and “means of storage”.

> Likewise, if you remembered every single word in a book, you would not understand its meaning

I understand your meaning but I want to clarify for the sake of the discussion that unlike with ML, the human brain can both memorize verbatim and understand the meaning because there is no mechanism for memorizing something but not processing it (i.e. purely storage). The first pass(es) are stripped to their essentials but subsequent passes provide the ability to memorize the same input.

Re: Do Machine Learning Models Memorize or Generalize?

#135

Post by [deleted] »

gattilorenz wrote: post #124
ComputerGuru wrote: post #45

Earlier quoted context omitted.

That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-bo…

Is there a “realistic upper bound” in things that should be memorized verbatim? Ancient greeks probably memorized the Iliad and other poems (rhyming and metre might work as a substitute for data compression, in this case), and many medieval preachers apparently memorized the whole Bible…

[deleted]

Re: Do Machine Learning Models Memorize or Generalize?

#136

Post by SanderNL »

ComputerGuru wrote: post #134
halflings wrote: post #82

Earlier quoted context omitted.

> That’s not exactly true [...] Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components ...but that's exactly what OP said, no? I remember attending an ML presentation where the speaker shared a quote I can't find anymore (speaking of memory and generalization :)), which said something like: "To learn is to forget" If we memorized everyt…

> but that's exactly what OP said, no? Not precisely. We don’t know if verbatim capacity is limited (and it doesn’t seem to be) but the brain operates in a space-efficient manner all the same. So there isn’t necessarily a causative relationship between “memory capacity” and “means of storage”. > Likewise, if you remembered every single word in a book, you would not understand its meaning I understand your meaning but…

We know for certain it is limited. Do brains not adhere to physics?

Re: Do Machine Learning Models Memorize or Generalize?

#137

Post by cmpalmer52 »

halflings wrote: post #82
ComputerGuru wrote: post #45

Earlier quoted context omitted.

That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-bo…

> That’s not exactly true [...] Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components ...but that's exactly what OP said, no? I remember attending an ML presentation where the speaker shared a quote I can't find anymore (speaking of memory and generalization :)), which said something like: "To learn is to forget" If we memorized everyt…

There’s a story by Jorge Luis Borges called “Funes the Memorious” about a man who remembers everything, but can’t generalize. There’s a line about him not knowing if a dog on the square glimpsed at noon from the side is the same dog as the one seen from the back at 12:01 or something like that. Swirls of smoke from a cigarette are memorized forever. He mostly sits in a dark room.

Re: Do Machine Learning Models Memorize or Generalize?

#139

Post by whimsicalism »

TheRealSteel wrote: post #131
ComputerGuru wrote: post #45

Earlier quoted context omitted.

That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-bo…

You seem to have just re-stated what the other person said.

Thank you, thought I was losing it for a second

Re: Do Machine Learning Models Memorize or Generalize?

#140

Post by whimsicalism »

ComputerGuru wrote: post #134
halflings wrote: post #82

Earlier quoted context omitted.

> That’s not exactly true [...] Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components ...but that's exactly what OP said, no? I remember attending an ML presentation where the speaker shared a quote I can't find anymore (speaking of memory and generalization :)), which said something like: "To learn is to forget" If we memorized everyt…

> but that's exactly what OP said, no? Not precisely. We don’t know if verbatim capacity is limited (and it doesn’t seem to be) but the brain operates in a space-efficient manner all the same. So there isn’t necessarily a causative relationship between “memory capacity” and “means of storage”. > Likewise, if you remembered every single word in a book, you would not understand its meaning I understand your meaning but…

> verbatim capacity is limited

I am but a simple physicist and I can already tell you it is.

Post reply on HN