Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixture-of-Depths: Dynamically allocating compute in transformers

arxiv.org

Post reply on HN
Print view View original

81–89 of 89 posts

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#81

Post by whimsicalism »

ska wrote: post #80
whimsicalism wrote: post #79

Earlier quoted context omitted.

The recursive neural networks described there are a failed academic project from more than a decade ago, predating modern deep learning. Basically everyone using the phrase recursive nn nowadays is probably just mispeaking for RNN. RNNs also are not linear

No opinion on the specifics of this distinction, but it's worth noting that in research, an awful lot of successful projects have their origins in failed projects of decades ago...

My experience working in machine learning academia is an overfocus on failed projects from the early 00s to 90s that really only stopped in 2020+.

We can often trace back successful projects to failed precursors, but often the people behind the successful project are not even familiar with the failed precursor and the 'connection to the past' only really occurs in retrospect. See the 'adjoint state method' and connections with backprop.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#82

Post by ska »

whimsicalism wrote: post #81
ska wrote: post #80

Earlier quoted context omitted.

No opinion on the specifics of this distinction, but it's worth noting that in research, an awful lot of successful projects have their origins in failed projects of decades ago...

My experience working in machine learning academia is an overfocus on failed projects from the early 00s to 90s that really only stopped in 2020+. We can often trace back successful projects to failed precursors, but often the people behind the successful project are not even familiar with the failed precursor and the 'connection to the past' only really occurs in retrospect. See the 'adjoint state method' and connec…

This is sometimes true, sure. And often the older work has more entered the general consciousness than being chased down by searching specific cites. On the other hand, very little is truly new, and recency bias can lead you into all sorts of back-eddy's.

Once the dust has settled, there are often much clearer through lines than in looked like at the time. It's hard to see when you are on the moving front though.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#83

Post by randcraw »

imtringued wrote: post #67
dannyw wrote: post #57

Earlier quoted context omitted.

I understand this is ELI5, but doesn’t attention already do this, in the way you described? It pays specific focus to the most contextual words in the prior sequence.

Not from a computational perspective. To calculate the attention score you have to calculate every token against every other token. That is quadratic. Every article like one, the, a, etc will have to be calculated against every other word even though they are only revelvant within a short distance of the word they are attached to.

Isn't that factorial, and much more costly than quadratic?

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#84

Post by visarga »

whimsicalism wrote: post #29
visarga wrote: post #19

Earlier quoted context omitted.

It routes values based on linear combinations taken from the attention map.

But all of those values are created using an MLP with the same parameters, so there is no routing to different parameters.

You have to look at it as a sequence of time steps which can interact. You can implement this interaction in many ways, such as transformer, mamba, rwkv or mlp-mixer. But the purpose is always to allow communication across time.

You use three distinct linear projections, one for queries, one for keys and one for values. From Q and K you compute the attention matrix A, and using A you construct linear combinations from V. But depending on A, for example for a token V_i there might be input from two other tokens, V_j or V_k, so information is moved between the tokens.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#85

Post by YeGoblynQueenne »

whimsicalism wrote: post #79
YeGoblynQueenne wrote: post #69

Earlier quoted context omitted.

Recursive NNs are not the same as Recurrent NNs: https://en.wikipedia.org/wiki/Recursive_neural_network Well ish. The article above explains that Recursive-NNs are hierarchical whereas RNNs are linear. I guess the distinction is a little on the fine side. Anyway carry on. Pedantic moment over.

The recursive neural networks described there are a failed academic project from more than a decade ago, predating modern deep learning. Basically everyone using the phrase recursive nn nowadays is probably just mispeaking for RNN. RNNs also are not linear

I don't know about "everybody nowadays" but I remember Recursive Neural Nets as an architecture introduced by Christopher Manning with the argument that it was better suited to the hierarchical structure of language than existing architectures. I did find it a bit of a bad choice of name, given that it's so closed to Recurrent Neural Nets. All this is from memory though I might check the internets later to see what I misremember.

RNNs are a large class of architectures of varying complexity, from Kallman Filters to LSTMs. It's not clear to me exactly what the wikipedia article means by "linear" but LSTMs for example treat their inputs as sequences and don't try to deconstruct them into parts, like e.g. Convolutional Neural Nets do. So maybe that's what's meant by "linear".

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#86

Post by dzamo_norton »

randcraw wrote: post #83
imtringued wrote: post #67

Earlier quoted context omitted.

Not from a computational perspective. To calculate the attention score you have to calculate every token against every other token. That is quadratic. Every article like one, the, a, etc will have to be calculated against every other word even though they are only revelvant within a short distance of the word they are attached to.

Isn't that factorial, and much more costly than quadratic?

N choose 2 = N! / 2!(N-2)! = N(N-1) / 2.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#87

Post by brrrrrm »

imtringued wrote: post #77
brrrrrm wrote: post #50

Earlier quoted context omitted.

It doesn’t. It simply trades compute efficiency by transposing matrix multiplications into “the future.” It doesn’t actually save FLOPs (uses more) and doesn’t work at large batch size

>doesn’t actually save FLOPs (uses more) Does anyone even care? Really, who cares? The truth is nobody cares. Saving FLOPs does nothing if you have to load the entire model anyway. Going from two flops per parameter to 0.5 or whatever might sound cool on paper but you're loading those parameters anyway and gained nothing.

companies that run these things care - they run at huge batch size and are compute bound in the limit

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#88

Post by mountainriver »

rajnathani wrote: post #52
mountainriver wrote: post #51

Earlier quoted context omitted.

Most of the original MoE implementations around LLMs were in fact recursive

Could you please elaborate?

The original MoE research done by Google around LLMs involved nested transformers to scale them. It was a layered approach where at each layer you would have set of experts, generally routed to by simple heuristics, then each of those models would call into its own series of experts and combine the data in various ways.

These models were SOTA for their time

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#89

Post by rajnathani »

mountainriver wrote: post #88
rajnathani wrote: post #52

Earlier quoted context omitted.

Could you please elaborate?

The original MoE research done by Google around LLMs involved nested transformers to scale them. It was a layered approach where at each layer you would have set of experts, generally routed to by simple heuristics, then each of those models would call into its own series of experts and combine the data in various ways. These models were SOTA for their time

Interesting, but that isn't recursive as the sub-model cannot invoke a model higher up in the invoke graph/tree.
Post reply on HN