Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The lottery ticket hypothesis: why neural networks work

nearlyright.com

Post reply on HN
Print view View original

51–60 of 69 posts

Re: The lottery ticket hypothesis: why neural networks work

#51

Post by ajuc »

physix wrote: post #43
mindwok wrote: post #35

Earlier quoted context omitted.

I'm no expert but have been thinking about this a lot lately. I wouldn't be surprised - language itself seems to be an expression of the ability to create an abstraction, distill the world into compressed representations, and manipulate symbols. It seems fundamental to human intelligence.

As a layman, it helps me to understand the importance of language as a vehicle of intelligence by realizing that without language, your thoughts are just emotions. And therefore I always thought that the more you master a language the better you are able to reason. And considering how much we let LLMs formulate text for us, how dumb will we get?

> without language, your thoughts are just emotions

That's not true. You can think "I want to walk around this building" without words, in abstract thoughts or in images.

Words are a layer above the thoughts, not the thoughts themselves. You can confirm this if you ever had the experience of trying to say something but forgetting the right word. Your mind knew what it wants to say but it didn't knew the word.

Chess players operate on sequences of moves dozen turns ahead in their minds using no words, seeing the moves on the virtual chessboards they imagine.

Musicians hear the note they want to play in their minds.

Our brains have full multimedia support.

Re: The lottery ticket hypothesis: why neural networks work

#52

Post by graemep »

pixl97 wrote: post #16
highfrequency wrote: post #8

Enjoyed the article. To play devil’s advocate, an entirely different explanation for why huge models work: the primary insight was framing the problem as next-word prediction. This immediately creates an internet-scale dataset with trillions of labeled examples, which also has rich enough structure to make huge expressiveness useful. LLMs don’t disprove bias-variance tradeoff; we just found a lot more data and the GP…

I think a lot of it is the massive amount of compute we've got in the last decade. While inference may have been possible on the hardware the training would have taken lifetimes.

I have a textbook somewhere in the house from about 2000 that says that there is no point having more than three layers in a neural network.

Compute was just too expensive to have neural networks big enough for this not to be true.

Re: The lottery ticket hypothesis: why neural networks work

#53

Post by leumassuehtam »

graemep wrote: post #52
pixl97 wrote: post #16

Earlier quoted context omitted.

I think a lot of it is the massive amount of compute we've got in the last decade. While inference may have been possible on the hardware the training would have taken lifetimes.

I have a textbook somewhere in the house from about 2000 that says that there is no point having more than three layers in a neural network. Compute was just too expensive to have neural networks big enough for this not to be true.

People believe that more parameters would lead to overfit instead generalization. The various regularization methods we use today to avoid overfit hadn't been discovered yet. Your statement is mostly likely about this.

Re: The lottery ticket hypothesis: why neural networks work

#54

Post by graemep »

leumassuehtam wrote: post #53
graemep wrote: post #52

Earlier quoted context omitted.

I have a textbook somewhere in the house from about 2000 that says that there is no point having more than three layers in a neural network. Compute was just too expensive to have neural networks big enough for this not to be true.

People believe that more parameters would lead to overfit instead generalization. The various regularization methods we use today to avoid overfit hadn't been discovered yet. Your statement is mostly likely about this.

Possibly, I would have to dig up the book to check. IIRC it did not mention overfitting but it was a long time ago.

Re: The lottery ticket hypothesis: why neural networks work

#56

Post by nitwit005 »

littlestymaar wrote: post #45
nitwit005 wrote: post #17

> For over 300 years, one principle governed every learning system This seems strangely worded. I assume that date is when some statistics paper was published, but there's no way to know with no definition or citations.

There is in fact a footnote about the date: > 1. The 300-year timeframe refers to the foundational mathematical principles underlying modern bias-variance analysis, not the contemporary terminology. Bayes' theorem (1763) established the mathematical framework for updating beliefs with evidence, whilst Laplace's early work on statistical inference (1780s-1810s) formalised the principle that models must balance fit wit…

They added it after my post.

Re: The lottery ticket hypothesis: why neural networks work

#57

Post by noboostforyou »

highfrequency wrote: post #48
deepsun wrote: post #39

Earlier quoted context omitted.

Same thing with Computer Vision, as Andrew Ng pointed out, the main thing that enabled the rapid progress was not new models, but mostly due to large _labeled_ datasets, particularly ImageNet.

Do you have a link handy for where he says this explicitly?

Here's any older interview where he talks about the need for accurate dataset labeling -

"In many industries where giant data sets simply don’t exist, I think the focus has to shift from big data to good data. Having 50 thoughtfully engineered examples can be sufficient to explain to the neural network what you want it to learn."

https://spectrum.ieee.org/andrew-ng-data-centric-ai

Re: The lottery ticket hypothesis: why neural networks work

#58

Post by Sophira »

mindwok wrote: post #35
unyttigfjelltol wrote: post #32

Earlier quoted context omitted.

Language then would be the key factor enabling complex learning in meat space too? I feel like I’ve heard this debate before….

I'm no expert but have been thinking about this a lot lately. I wouldn't be surprised - language itself seems to be an expression of the ability to create an abstraction, distill the world into compressed representations, and manipulate symbols. It seems fundamental to human intelligence.

In other words, we're rediscovering the lessons from George Orwell's Nineteen Eighty-Four. Language is central to understanding; remove subversive language and you remove the ability to even think about it.

Re: The lottery ticket hypothesis: why neural networks work

#59

Post by Sophira »

janalsncm wrote: post #27
xg15 wrote: post #4

Wouldn't this imply that most of the inference time storage and compute might be unnecessary? If the hypothesis is true, it makes sense to scale up models as much as possible during training - but once the model is sufficiently trained for the task, wouldn't 99% of the weights be literal "dead weight" - because they represent the "failed lottery tickets", i.e. the subnetworks that did not have the right starting valu…

Quick example, Kimi K2 is a recent large mixture of experts model. Each “expert” is really just a path within it. At each token, 32B out of 1T are active. This means only 3.2% are active for any one token.

That sounds surprisingly like "Humans only use 10% of their brain at any given time."

Re: The lottery ticket hypothesis: why neural networks work

#60

Post by ActorNightly »

math_dandy wrote: post #30

I don't buy the narrative that the article is promoting. I think the machine learning community was largely over overfitophobia by 2019 and people were routinely using overparametrized models capable of interpolating their training data while still generalizing well. The Belkin et al. paper wasn't heresy. The authors were making a technical point - that certain theories of generalization are incompatible with this in…

Yeah this article gets a whole bunch of history wrong.

Back in 2000s, the reason why nobody was pursuing neural nets was simply due to compute power, and the fact that you couldn't iterate fast enough to make smaller neural networks work.

People were doing genetic algorithms and PSO for quite some time. Everyone knew that multi dimentionality was the solution to overfitting - the more directions you can use to climb out of valleys the better the system performed.

Post reply on HN