Mixture-of-Depths: Dynamically allocating compute in transformers
1–10 of 89 posts
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#2It’s very similar to Mixture of Experts. But instead of routing tokens to multiple experts, you "deploy to a single expert which can be dynamically skipped"
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#3[deleted]
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#4I think more complicated routing is absolutely going to become more common.
Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#5I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
We can name these hypothetical objects Recursive Neural Networks.
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#6It’s very similar to Mixture of Experts. But instead of routing tokens to multiple experts, you "deploy to a single expert which can be dynamically skipped"
Mixing these would be pretty cool. Further reduced compute for MoE while keeping the performance.
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#7It’s very similar to Mixture of Experts. But instead of routing tokens to multiple experts, you "deploy to a single expert which can be dynamically skipped"
Mixing these would be pretty cool. Further reduced compute for MoE while keeping the performance.
In the paper they already show a mixing of these two with Mixture-of-Depths-and-Experts (MoDE).
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#8I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
See, this is where my understanding of LLMs breaks down. I can understand one token going through the model, but I can't understand a model that has different "experts" internally.
Do you have any resources or links to help explain that concept?
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#9I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
See, this is where my understanding of LLMs breaks down. I can understand one token going through the model, but I can't understand a model that has different "experts" internally. Do you have any resources or links to help explain that concept?
It is still just one token going through the model.
I actually think mixture-of-expert is a bit of a misnomer, the 'experts' do not really necessarily have super distinct expertise. Think of it more as how neurons activate in the brain - your entire brain doesn't light up for every query, now in neural networks the same thing happens (it doesn't fully light up for every query).
Don't really know a resource besides the seminal Noam Shazeer paper, sorry - I'm sure others have higher-level.
Re: Mixture-of-Depths: Dynamically allocating compute in transformers
#10I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
We can name these hypothetical objects Recursive Neural Networks.
i know you're jesting but RNNs are recursive along the sequence length where I am describing recursion along the depth.