Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixture-of-Depths: Dynamically allocating compute in transformers

arxiv.org

Post reply on HN
Print view View original

21–30 of 89 posts

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#21

Post by pizza »

londons_explore wrote: post #17
whimsicalism wrote: post #4

I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively

I think the reason this hasn't been done is you have no way to decide how many recursions are necessary at train time. And if you pick a random number/try many different levels of recursion, you 'blur' the output. Ie. the output of a layer doesn't know if it should be outputting info important for the final result, or the output that is the best possible input to another round of recursion.

You could just learn the right estimated number of recursions, also passing 'backtracking'/'state' information at the next nested level. Kind of like how state space models encode extractible information via a basis function representation, you could encode extractible recursion state information into the embedding. See also transformers that can learn to recognize n-deep balanced parentheses (Dyck-n languages)

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#22

Post by visarga »

ein0p wrote: post #14
barrenko wrote: post #11

Are we going to hit bullseye?

This only cuts compute by “up to” 50% and only during inference. Quadratic dependence on context size remains, as do the enormous memory requirements. For something to be considered a bulls eye in this space it has to offer nonlinear improvements on both of these axes, and/or be much faster to train. Until that happens, people, including Google will continue to train bog standard MoE and dense transformers. Radical e…

Yeah all attempts at reducing complexity from quadratic to linear failed, only Mamba still has a chance, but it's not tested on large models and only provides a speedup at for 2000+ tokens. That was to be expected as small sequences have very small memory requirements for transformers, but recursive architectures use the same hidden size. So when recurrent hidden size > sequence length, the old transformer is faster.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#23

Post by refulgentis »

whimsicalism wrote: post #10
optimalsolver wrote: post #5

Earlier quoted context omitted.

We can name these hypothetical objects Recursive Neural Networks.

i know you're jesting but RNNs are recursive along the sequence length where I am describing recursion along the depth.

Like decode the next token, then adjust what you're paying attention to, then decode it again?

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#24

Post by mdale »

ein0p wrote: post #14
barrenko wrote: post #11

Are we going to hit bullseye?

This only cuts compute by “up to” 50% and only during inference. Quadratic dependence on context size remains, as do the enormous memory requirements. For something to be considered a bulls eye in this space it has to offer nonlinear improvements on both of these axes, and/or be much faster to train. Until that happens, people, including Google will continue to train bog standard MoE and dense transformers. Radical e…

Makes opportunities for smaller companies to innovative/experiment to offer solutions / acquisition targets where tighter inference compute requirements makes or breaks the experience but larger training cost is less of a concern (such as embedded or local runtime use cases)

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#25

Post by conradev »

optimalsolver wrote: post #5
whimsicalism wrote: post #4

I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively

We can name these hypothetical objects Recursive Neural Networks.

Yep: https://arxiv.org/abs/2305.13048

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#26

Post by ein0p »

visarga wrote: post #22
ein0p wrote: post #14

Earlier quoted context omitted.

This only cuts compute by “up to” 50% and only during inference. Quadratic dependence on context size remains, as do the enormous memory requirements. For something to be considered a bulls eye in this space it has to offer nonlinear improvements on both of these axes, and/or be much faster to train. Until that happens, people, including Google will continue to train bog standard MoE and dense transformers. Radical e…

Yeah all attempts at reducing complexity from quadratic to linear failed, only Mamba still has a chance, but it's not tested on large models and only provides a speedup at for 2000+ tokens. That was to be expected as small sequences have very small memory requirements for transformers, but recursive architectures use the same hidden size. So when recurrent hidden size > sequence length, the old transformer is faster.

It's more subtle than that IMO. They haven't necessarily "failed" - they just don't have the "superpowers" that the metrics used to evaluate such systems are aimed at. E.g. no such linear method devised so far (that I know of, at least) is able to do very high recall point retrieval in long context _and_ effective in-context learning simultaneously. You get one or the other, but not both. But as far as the metrics go, high recall retrieval in long context is easier to for the researcher to demonstrate and for the observer to comprehend - a typical needle/haystack setting is trivial to put together. It is also something that (unlike in-context learning) humans are usually very bad at, so it's perceived as a "superpower" or "magic". In this case e.g. Mamba being more human like due to its selective forgetfulness is currently playing against it. But whether it's "better" per se will depend on the task. It's just that we do not know how to evaluate most of the tasks yet, so people keep trying to find the proverbial keys under the lamp post, and measure what they can to make progress, and thereby keep their efforts lavishly funded.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#27

Post by ein0p »

mdale wrote: post #24
ein0p wrote: post #14

Earlier quoted context omitted.

This only cuts compute by “up to” 50% and only during inference. Quadratic dependence on context size remains, as do the enormous memory requirements. For something to be considered a bulls eye in this space it has to offer nonlinear improvements on both of these axes, and/or be much faster to train. Until that happens, people, including Google will continue to train bog standard MoE and dense transformers. Radical e…

Makes opportunities for smaller companies to innovative/experiment to offer solutions / acquisition targets where tighter inference compute requirements makes or breaks the experience but larger training cost is less of a concern (such as embedded or local runtime use cases)

Before those opportunities are available to you, someone would need to spend a few million dollars and train a competitive model with this, and then release it under a license that allows commercial use. This is out of reach for the vast majority of smaller companies. These models only excel at large parameter counts, even for narrow problems. This is especially true in the case of MoE, which is a way to push the overall parameter count even larger without lighting up the whole thing for every token.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#28

Post by [deleted] »

refulgentis wrote: post #23
whimsicalism wrote: post #10

Earlier quoted context omitted.

i know you're jesting but RNNs are recursive along the sequence length where I am describing recursion along the depth.

Like decode the next token, then adjust what you're paying attention to, then decode it again?

[deleted]

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#29

Post by whimsicalism »

visarga wrote: post #19
whimsicalism wrote: post #16

Earlier quoted context omitted.

How is attention basically routing?

It routes values based on linear combinations taken from the attention map.

But all of those values are created using an MLP with the same parameters, so there is no routing to different parameters.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#30

Post by sdenton4 »

londons_explore wrote: post #17
whimsicalism wrote: post #4

I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively

I think the reason this hasn't been done is you have no way to decide how many recursions are necessary at train time. And if you pick a random number/try many different levels of recursion, you 'blur' the output. Ie. the output of a layer doesn't know if it should be outputting info important for the final result, or the output that is the best possible input to another round of recursion.

This is actually how EfficientNet trains, using random truncation of the network during training. It does just fine... The game is that each layer needs to get as close as it can to good output, improving in the previous activation quality.
Post reply on HN