Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The lottery ticket hypothesis: why neural networks work

nearlyright.com

Post reply on HN
Print view View original

11–20 of 69 posts

Re: The lottery ticket hypothesis: why neural networks work

#11

Post by paulsutter »

xg15 wrote: post #4

Wouldn't this imply that most of the inference time storage and compute might be unnecessary? If the hypothesis is true, it makes sense to scale up models as much as possible during training - but once the model is sufficiently trained for the task, wouldn't 99% of the weights be literal "dead weight" - because they represent the "failed lottery tickets", i.e. the subnetworks that did not have the right starting valu…

That’s exactly how it works, read up on pruning. You can ignore most of the weights and still get great results. One issue is that sparse matrices are vastly less efficient to multiply.

But yes you’ve got it

Re: The lottery ticket hypothesis: why neural networks work

#15

Post by FuckButtons »

xg15 wrote: post #4

Wouldn't this imply that most of the inference time storage and compute might be unnecessary? If the hypothesis is true, it makes sense to scale up models as much as possible during training - but once the model is sufficiently trained for the task, wouldn't 99% of the weights be literal "dead weight" - because they represent the "failed lottery tickets", i.e. the subnetworks that did not have the right starting valu…

For any particular single pattern learned 99% of the weights are dead weight. But it’s not the same 99% for each lesson learned.

Re: The lottery ticket hypothesis: why neural networks work

#16

Post by pixl97 »

highfrequency wrote: post #8

Enjoyed the article. To play devil’s advocate, an entirely different explanation for why huge models work: the primary insight was framing the problem as next-word prediction. This immediately creates an internet-scale dataset with trillions of labeled examples, which also has rich enough structure to make huge expressiveness useful. LLMs don’t disprove bias-variance tradeoff; we just found a lot more data and the GP…

I think a lot of it is the massive amount of compute we've got in the last decade. While inference may have been possible on the hardware the training would have taken lifetimes.

Re: The lottery ticket hypothesis: why neural networks work

#18

Post by doctoboggan »

This article definitely feels like chatgptese.

Also, I don't necessarily feel like the size of LLMs even comes close to overfitting the data. From a very unscientific standpoint it seems like the size of weights on disk would have to meet or exceed the size of the training data (modulo lossless encryption techniques) for overfitting to occur. Since the training data is multiple orders of magnitude larger than the resulting weights, isn't that proof that the weights are some sort of generalization of the input data rather than a memorization?

Re: The lottery ticket hypothesis: why neural networks work

#19

Post by pixl97 »

deepfriedchokes wrote: post #12

Rather than reframing intelligence itself, wouldn’t Occam’s Razor suggest instead that this isn’t intelligence at all?

I don't really think that is what Occam’s Razor is about. The Razor says the simplest answer is most likely the best, but we already know that intelligence is very complex so the simplest answer to intelligence is still going to be a massively complex solution.

In some ways this answer does fit Occam's Razor by saying the simplicity is simply scale, not complex algorithms.

Re: The lottery ticket hypothesis: why neural networks work

#20

Post by anthonj »

ghssds wrote: post #14

Can someone explain how AI research can have a 300 years history?

"For over 300 years, one principle governed every learning system: the bias-variance tradeoff."

The bias-variance tradeoff is a very old concept in statistics (but not sure how old, might very well be 300)

Anyway note the first algorithms realted to neural networks are older then the digital computer by a decade at least.

Post reply on HN