Are we going to hit bullseye?
Mixture-of-Depths: Dynamically allocating compute in transformers
11–20 of 89 posts
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#12I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
What you describe here sounds a little like the line of work centered around Universal Transformers, which basically process the input embeddings through a single transformer block multiple times with a separate module deciding when the embeddings have been cooked enough and can be pulled out of the oven so to speak.
Even more in line with the idea of "experts" there's a paper from last year on Sparse Universal Transformers in which they combine a universal transformer with sparse mixture of experts, so it's up to the gating mechanism to decide which transformer blocks and in which order are to be used in shaping the embeddings.
This really isn't my specialty but from what I gathered these are tricky to train properly, and require more overall compute during inference to reach comparable results to their vanilla transformer counterparts. It's an interesting direction nonetheless, having an upper bound on the number of computation steps per token is, in my opinion, one of the major downsides of the classical transformer architecture.
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#13I wrote up a bit about it here, from what I could piece together:
https://lifeinthesingularity.com/p/googles-breakthroughs-in-...
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#14Are we going to hit bullseye?
This only cuts compute by “up to” 50% and only during inference. Quadratic dependence on context size remains, as do the enormous memory requirements. For something to be considered a bulls eye in this space it has to offer nonlinear improvements on both of these axes, and/or be much faster to train. Until that happens, people, including Google will continue to train bog standard MoE and dense transformers. Radical experimentation at scale is too expensive even for megacorps at this point.
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#15I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
Attention is basically routing, these other routing schemes put a less fine-grained choice for the model, which potentially makes it easier to train
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#16I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
Attention is basically routing, these other routing schemes put a less fine-grained choice for the model, which potentially makes it easier to train
How is attention basically routing?
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#17I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
I think the reason this hasn't been done is you have no way to decide how many recursions are necessary at train time.
And if you pick a random number/try many different levels of recursion, you 'blur' the output. Ie. the output of a layer doesn't know if it should be outputting info important for the final result, or the output that is the best possible input to another round of recursion.
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#18I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
I think the reason this hasn't been done is you have no way to decide how many recursions are necessary at train time. And if you pick a random number/try many different levels of recursion, you 'blur' the output. Ie. the output of a layer doesn't know if it should be outputting info important for the final result, or the output that is the best possible input to another round of recursion.
Yes, I think training this model would be hard. Perhaps something akin to how MoEs are trained where you impose some sort of loss distribution to encourage equitable routing, but for recursion.
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#19Earlier quoted context omitted.
Attention is basically routing, these other routing schemes put a less fine-grained choice for the model, which potentially makes it easier to train
How is attention basically routing?
It routes values based on linear combinations taken from the attention map.
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#20[dead]