Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The lottery ticket hypothesis: why neural networks work

nearlyright.com

Post reply on HN
Print view View original

41–50 of 69 posts

Re: The lottery ticket hypothesis: why neural networks work

#41

Post by krackers »

Eisenstein wrote: post #21
highfrequency wrote: post #8

Enjoyed the article. To play devil’s advocate, an entirely different explanation for why huge models work: the primary insight was framing the problem as next-word prediction. This immediately creates an internet-scale dataset with trillions of labeled examples, which also has rich enough structure to make huge expressiveness useful. LLMs don’t disprove bias-variance tradeoff; we just found a lot more data and the GP…

Why does 'next-word prediction' explain why huge models work? You saying we needed scale, and saying we use next-word prediction, but how does one relate to the other? Diffusion models also exist and work well for images, and they do seem to work for LLMs too.

I think it's the same underlying principle of learning the "joint distribution of things humans have said". Whether done autoregressively via LLMs or via diffusion models, you still end up learning this distribution. The insight seems to be the crazy leap that this is A) a valid thing to talk about and B) that learning this distribution gives you something meaningful.

The leap is in transforming an ill-defined objective of "modeling intelligence" into a concrete proxy objective. Note that the task isn't even "distribution set of valid/true things", since validity/truth is hard to define. It's something akin to "distribution of things a human might say" implemented in the "dumbest" possible way of "modeling the distribution of humanity's collective textual output".

Re: The lottery ticket hypothesis: why neural networks work

#42

Post by porridgeraisin »

doctoboggan wrote: post #18

This article definitely feels like chatgptese. Also, I don't necessarily feel like the size of LLMs even comes close to overfitting the data. From a very unscientific standpoint it seems like the size of weights on disk would have to meet or exceed the size of the training data (modulo lossless encryption techniques) for overfitting to occur. Since the training data is multiple orders of magnitude larger than the res…

1) yes it's definitely chatgpt

2) The weights are definitely a generalization. The compression-based argument is sound.

3) There is definitely no overfitting. The article however used the word over-parameterization, which is a different thing. And LLMs are certainly over-parameterized. They have more parameters than strictly required to represent the dataset in a degrees-of-freedom statistical sense. This is not a bad thing though.

Just like having an over-parameterized database schema:

  quiz(id, title, num_qns)
  question(id, text, answer, quiz_id FK)
can be good for performance sometimes,

The lottery ticket hypothesis as chatgpt explained in TFA means that over-parameterization can also be good for neural networks sometimes. Note that this hypothesis is strictly tied to the fact that we use SGD (or adam or ...) as the optimisation algorithm. SGD is known to be biased towards generalized compressions [the lottery ticket hypothesis hypothesises why this is so]. That is to say, it's not an inherent property of the neural network architecture or transformers or such.

Re: The lottery ticket hypothesis: why neural networks work

#43

Post by physix »

mindwok wrote: post #35
unyttigfjelltol wrote: post #32

Earlier quoted context omitted.

Language then would be the key factor enabling complex learning in meat space too? I feel like I’ve heard this debate before….

I'm no expert but have been thinking about this a lot lately. I wouldn't be surprised - language itself seems to be an expression of the ability to create an abstraction, distill the world into compressed representations, and manipulate symbols. It seems fundamental to human intelligence.

As a layman, it helps me to understand the importance of language as a vehicle of intelligence by realizing that without language, your thoughts are just emotions.

And therefore I always thought that the more you master a language the better you are able to reason.

And considering how much we let LLMs formulate text for us, how dumb will we get?

Re: The lottery ticket hypothesis: why neural networks work

#44

Post by [deleted] »

krackers wrote: post #41
Eisenstein wrote: post #21

Earlier quoted context omitted.

Why does 'next-word prediction' explain why huge models work? You saying we needed scale, and saying we use next-word prediction, but how does one relate to the other? Diffusion models also exist and work well for images, and they do seem to work for LLMs too.

I think it's the same underlying principle of learning the "joint distribution of things humans have said". Whether done autoregressively via LLMs or via diffusion models, you still end up learning this distribution. The insight seems to be the crazy leap that this is A) a valid thing to talk about and B) that learning this distribution gives you something meaningful. The leap is in transforming an ill-defined object…

[deleted]

Re: The lottery ticket hypothesis: why neural networks work

#45

Post by littlestymaar »

nitwit005 wrote: post #17

> For over 300 years, one principle governed every learning system This seems strangely worded. I assume that date is when some statistics paper was published, but there's no way to know with no definition or citations.

There is in fact a footnote about the date:

> 1. The 300-year timeframe refers to the foundational mathematical principles underlying modern bias-variance analysis, not the contemporary terminology. Bayes' theorem (1763) established the mathematical framework for updating beliefs with evidence, whilst Laplace's early work on statistical inference (1780s-1810s) formalised the principle that models must balance fit with simplicity to avoid spurious conclusions. These early statistical insights—that overly complex explanations tend to capture noise rather than signal—form the mathematical bedrock of what we now call the bias-variance tradeoff. The specific modern formulation emerged over several decades in the latter 20th century, but the core principle has governed statistical reasoning for centuries.

Re: The lottery ticket hypothesis: why neural networks work

#46

Post by littlestymaar »

ghssds wrote: post #14

Can someone explain how AI research can have a 300 years history?

Maybe it wasn't there originally, but now there's a footnote:

> 1. The 300-year timeframe refers to the foundational mathematical principles underlying modern bias-variance analysis, not the contemporary terminology. Bayes' theorem (1763) established the mathematical framework for updating beliefs with evidence, whilst Laplace's early work on statistical inference (1780s-1810s) formalised the principle that models must balance fit with simplicity to avoid spurious conclusions. These early statistical insights—that overly complex explanations tend to capture noise rather than signal—form the mathematical bedrock of what we now call the bias-variance tradeoff. The specific modern formulation emerged over several decades in the latter 20th century, but the core principle has governed statistical reasoning for centuries.↩

Re: The lottery ticket hypothesis: why neural networks work

#47

Post by xg15 »

unyttigfjelltol wrote: post #32
highfrequency wrote: post #8

Enjoyed the article. To play devil’s advocate, an entirely different explanation for why huge models work: the primary insight was framing the problem as next-word prediction. This immediately creates an internet-scale dataset with trillions of labeled examples, which also has rich enough structure to make huge expressiveness useful. LLMs don’t disprove bias-variance tradeoff; we just found a lot more data and the GP…

Language then would be the key factor enabling complex learning in meat space too? I feel like I’ve heard this debate before….

I think it doesn't have to follow. You could also generalize the idea and see learning as successfully being able to "use the past to predict the future" for small time increments. Next-word prediction would be one instance of this, but for humans and animals, you could imagine the same process with information from all senses. The "self-supervised" trainset is then just, well, life.

Re: The lottery ticket hypothesis: why neural networks work

#48

Post by highfrequency »

deepsun wrote: post #39
highfrequency wrote: post #8

Enjoyed the article. To play devil’s advocate, an entirely different explanation for why huge models work: the primary insight was framing the problem as next-word prediction. This immediately creates an internet-scale dataset with trillions of labeled examples, which also has rich enough structure to make huge expressiveness useful. LLMs don’t disprove bias-variance tradeoff; we just found a lot more data and the GP…

Same thing with Computer Vision, as Andrew Ng pointed out, the main thing that enabled the rapid progress was not new models, but mostly due to large _labeled_ datasets, particularly ImageNet.

Do you have a link handy for where he says this explicitly?

Re: The lottery ticket hypothesis: why neural networks work

#49

Post by highfrequency »

physix wrote: post #43
mindwok wrote: post #35

Earlier quoted context omitted.

I'm no expert but have been thinking about this a lot lately. I wouldn't be surprised - language itself seems to be an expression of the ability to create an abstraction, distill the world into compressed representations, and manipulate symbols. It seems fundamental to human intelligence.

As a layman, it helps me to understand the importance of language as a vehicle of intelligence by realizing that without language, your thoughts are just emotions. And therefore I always thought that the more you master a language the better you are able to reason. And considering how much we let LLMs formulate text for us, how dumb will we get?

> without language, your thoughts are just emotions.

Is that true though? Seems like you can easily have some cognitive process that visualizes things like cause and effect, simple algorithms or at least sequences of events.

Re: The lottery ticket hypothesis: why neural networks work

#50

Post by highfrequency »

Eisenstein wrote: post #21
highfrequency wrote: post #8

Enjoyed the article. To play devil’s advocate, an entirely different explanation for why huge models work: the primary insight was framing the problem as next-word prediction. This immediately creates an internet-scale dataset with trillions of labeled examples, which also has rich enough structure to make huge expressiveness useful. LLMs don’t disprove bias-variance tradeoff; we just found a lot more data and the GP…

Why does 'next-word prediction' explain why huge models work? You saying we needed scale, and saying we use next-word prediction, but how does one relate to the other? Diffusion models also exist and work well for images, and they do seem to work for LLMs too.

To crack NLP we needed a large dataset of labeled language examples. Prior to next-word prediction, the dominant benchmarks and datasets were things like translation of English to German sentences. These datasets were on the order of millions of labeled examples. Next-word prediction turned the entire Internet into labeled data.
Post reply on HN