[deleted]
The lottery ticket hypothesis: why neural networks work
61–69 of 69 posts
Re: The lottery ticket hypothesis: why neural networks work
#62Enjoyed the article. To play devil’s advocate, an entirely different explanation for why huge models work: the primary insight was framing the problem as next-word prediction. This immediately creates an internet-scale dataset with trillions of labeled examples, which also has rich enough structure to make huge expressiveness useful. LLMs don’t disprove bias-variance tradeoff; we just found a lot more data and the GP…
But regarding this Lottery Ticket hypothesis, what it means is that a small percentage of the parameters can be identified such that: when those parameters are taken by themselves and reset to their original pre-training weights, and the resulting network is trained on the same data as the parent, it performs similarly to the parent. So in fact, it seems that far fewer parameters are needed to encode predictions across the Internet-scale dataset. The large model is just creating a space in which that small "rock star" subset of parameters can be automatically discovered. It's as if the training establishes a competition among small subsets of the network, where a winner emerges.
Perhaps there is a kind of unstable situation whereby once the winner starts to anneal toward predicting the training data, it is doing more and more of the predictive work. The more relevant the subset shows itself to the result, the more of the learning it captures, because it is more responsive to subsequent learning.
Re: The lottery ticket hypothesis: why neural networks work
#63Enjoyed the article. To play devil’s advocate, an entirely different explanation for why huge models work: the primary insight was framing the problem as next-word prediction. This immediately creates an internet-scale dataset with trillions of labeled examples, which also has rich enough structure to make huge expressiveness useful. LLMs don’t disprove bias-variance tradeoff; we just found a lot more data and the GP…
Language then would be the key factor enabling complex learning in meat space too? I feel like I’ve heard this debate before….
I think that the takeaway message for meat space (if there is one) is that continuous life-long learning is where it is at: keep engaging your brain and playing the lottery in order to foster the winning tickets. Be exposed to a variety of stimuli and find relationships.
Re: The lottery ticket hypothesis: why neural networks work
#64Earlier quoted context omitted.
I have a textbook somewhere in the house from about 2000 that says that there is no point having more than three layers in a neural network. Compute was just too expensive to have neural networks big enough for this not to be true.
People believe that more parameters would lead to overfit instead generalization. The various regularization methods we use today to avoid overfit hadn't been discovered yet. Your statement is mostly likely about this.
I think the problems with big network were diminishing gradients, which is why we now use the ReLU activation function, and training stability, which were solved with residual connections.
Overfitting is the problem of having too little training data for your network size.
Re: The lottery ticket hypothesis: why neural networks work
#65This article gives a really bad/wrong explanation of the lottery ticket hypothesis. Here's the original paper https://arxiv.org/abs/1803.03635
Thanks for the 42 page long document. Can you explain in few words why you evaluated it as "really bad/wrong explanation"?
What are LLMs for?
Re: The lottery ticket hypothesis: why neural networks work
#66Earlier quoted context omitted.
I think a lot of it is the massive amount of compute we've got in the last decade. While inference may have been possible on the hardware the training would have taken lifetimes.
I have a textbook somewhere in the house from about 2000 that says that there is no point having more than three layers in a neural network. Compute was just too expensive to have neural networks big enough for this not to be true.
Once you have three layers (i.e. one "hidden" layer) then you can map to arbitrary functions, so a three layer network has the same "power" as an arbitrarily large network.
I'm sure that's what the text book meant, rather than any point about the expense of computing power.
Re: The lottery ticket hypothesis: why neural networks work
#67Earlier quoted context omitted.
As a layman, it helps me to understand the importance of language as a vehicle of intelligence by realizing that without language, your thoughts are just emotions. And therefore I always thought that the more you master a language the better you are able to reason. And considering how much we let LLMs formulate text for us, how dumb will we get?
> without language, your thoughts are just emotions That's not true. You can think "I want to walk around this building" without words, in abstract thoughts or in images. Words are a layer above the thoughts, not the thoughts themselves. You can confirm this if you ever had the experience of trying to say something but forgetting the right word. Your mind knew what it wants to say but it didn't knew the word. Chess p…
It's probably not as simple as just being emotions, but actually there's a really interesting example here: Helen Keller. In her autobiography she describes what it was like before she learned language, and how she remembers it being almost unconscious and just a mix of feelings and impulses. It's fascinating.
Re: The lottery ticket hypothesis: why neural networks work
#68This article is like a quick street rap. Lots of rhythm, not much thesis. Big on tone, light on analysis...Or no actual thesis other than a feelgood factor. I want these 5 min back.
> I want these 5 min back.
Tell me, what is it you plan to do
with your five wild and precious minutes?
Re: The lottery ticket hypothesis: why neural networks work
#69Isn’t small or large in relation to the amount of data, and the current large models a result of there being so incredibly much data available?