Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixture-of-Depths: Dynamically allocating compute in transformers

arxiv.org

Post reply on HN
Print view View original

61–70 of 89 posts

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#61

Post by hackerlight »

whimsicalism wrote: post #18
londons_explore wrote: post #17

Earlier quoted context omitted.

I think the reason this hasn't been done is you have no way to decide how many recursions are necessary at train time. And if you pick a random number/try many different levels of recursion, you 'blur' the output. Ie. the output of a layer doesn't know if it should be outputting info important for the final result, or the output that is the best possible input to another round of recursion.

Yes, I think training this model would be hard. Perhaps something akin to how MoEs are trained where you impose some sort of loss distribution to encourage equitable routing, but for recursion.

Look at the human brain for useful analogies?

The default mode network does recursive/looping processing in the absence of external stimuli and world interaction. Multiple separate modules outside of the network are responsible for stopping and regulating this activity.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#63

Post by namibj »

nl wrote: post #58

Most important paper of 2024. The idea that we want models not to have to use the same amount of compute for every token has been around for a while. This is the first compelling mechanism I've seen for doing it. > Equipped with these new methods, we can sample autoregressively by choosing to route tokens to or around a block based on the router’s output, which does not depend on any information from future tokens. W…

Sparse Universal Transformer is older and already did routing-based early termination...

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#64

Post by macrolime »

"This is more computationally efficient than performing a full content-based lookup across an entire memory buffer for each step in the future, and could be one step towards drastically increasing the context-length available for making a prediction."

Is this how they get a context window of 10 million tokens? Or are they refering to even longer context windows in the future?

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#65

Post by Nesco »

londons_explore wrote: post #17
whimsicalism wrote: post #4

I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively

I think the reason this hasn't been done is you have no way to decide how many recursions are necessary at train time. And if you pick a random number/try many different levels of recursion, you 'blur' the output. Ie. the output of a layer doesn't know if it should be outputting info important for the final result, or the output that is the best possible input to another round of recursion.

I have been thinking about this topic for some time. It might be done using the energy of the token. If it's still higher than an energy limit, then process it again, and increase the energy limit. The energy could be computed using log-sum-exp: https://openreview.net/pdf?id=Hkxzx0NtDB

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#66

Post by imtringued »

nl wrote: post #58

Most important paper of 2024. The idea that we want models not to have to use the same amount of compute for every token has been around for a while. This is the first compelling mechanism I've seen for doing it. > Equipped with these new methods, we can sample autoregressively by choosing to route tokens to or around a block based on the router’s output, which does not depend on any information from future tokens. W…

Most important? The idea that not every token needs the full context window should be an obvious optimization.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#67

Post by imtringued »

dannyw wrote: post #57
panqueca wrote: post #34

Simplified Intro Version: Imagine you have a smart assistant that can understand and process the words you say to it. Usually, this assistant pays equal attention to every word you say, no matter how important or unimportant each word is to the overall meaning of your message. Now, imagine that we found a way to teach the assistant to be smarter about how it uses its "brain power." Instead of giving equal attention t…

I understand this is ELI5, but doesn’t attention already do this, in the way you described? It pays specific focus to the most contextual words in the prior sequence.

Not from a computational perspective. To calculate the attention score you have to calculate every token against every other token. That is quadratic. Every article like one, the, a, etc will have to be calculated against every other word even though they are only revelvant within a short distance of the word they are attached to.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#69

Post by YeGoblynQueenne »

whimsicalism wrote: post #10
optimalsolver wrote: post #5

Earlier quoted context omitted.

We can name these hypothetical objects Recursive Neural Networks.

i know you're jesting but RNNs are recursive along the sequence length where I am describing recursion along the depth.

Recursive NNs are not the same as Recurrent NNs:

https://en.wikipedia.org/wiki/Recursive_neural_network

Well ish. The article above explains that Recursive-NNs are hierarchical whereas RNNs are linear. I guess the distinction is a little on the fine side.

Anyway carry on. Pedantic moment over.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#70

Post by cosmojg »

whimsicalism wrote: post #4

I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively

There are already some implementations out there which attempt to accomplish this!

Here's an example: https://github.com/silphendio/sliced_llama

A gist pertaining to said example: https://gist.github.com/silphendio/535cd9c1821aa1290aa10d587...

Here's a discussion about integrating this capability with ExLlama: https://github.com/turboderp/exllamav2/pull/275

And same as above but for llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4718#issuecomm...

Post reply on HN