Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixture-of-Depths: Dynamically allocating compute in transformers

arxiv.org

Post reply on HN
Print view View original

41–50 of 89 posts

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#41

Post by HarHarVeryFunny »

digdugdirk wrote: post #8
whimsicalism wrote: post #4

I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively

See, this is where my understanding of LLMs breaks down. I can understand one token going through the model, but I can't understand a model that has different "experts" internally. Do you have any resources or links to help explain that concept?

The "mixture of experts" goal is to add more parameters to the model to make it more powerful, without requiring any more compute. The way this is done is by having sections of the model ("experts") that are in parallel with each other, and each token only going through one of them. Think of it like a multi-lane highway with a toll booth on each lane - each car only drives on one lane rather than using them all, so only pays one toll.

The name "experts" is a bit misleading, since each expert ("highway lane") is not really specialized in any obviously meaningful way. There is a routing/gating component in front of the experts that chooses on a token by token basis (not sentence by sentence!) which "expert" to route the token to, with the goal of roughly load balancing between the experts so that they all see the same number of tokens, and the parameters in each expert are therefore all equally utilized.

The fact that the tokens in a sentence will be somewhat arbitrarily sent through different "experts" makes it an odd kind of expertise - not directly related to the sentence as a whole! There has been experimentation with a whole bunch of routing (expert selection) schemes.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#42

Post by [deleted] »

panqueca wrote: post #34

Simplified Intro Version: Imagine you have a smart assistant that can understand and process the words you say to it. Usually, this assistant pays equal attention to every word you say, no matter how important or unimportant each word is to the overall meaning of your message. Now, imagine that we found a way to teach the assistant to be smarter about how it uses its "brain power." Instead of giving equal attention t…

[deleted]

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#44

Post by rfdearborn »

whimsicalism wrote: post #4

I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively

The trendline is definitely toward increasing dynamic routing, but I suspect it's more so that MoE/MoD/MoDE enable models to embed additional facts with less superposition within their weights than enable deeper reasoning. Instead I expect deeper reasoning will come through token-wise dynamism rather than layer-wise -- e.g., this recent Quiet-STaR paper in which the model outputs throwaway rationale tokens: https://arxiv.org/abs/2403.09629

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#45

Post by datascienced »

mattmcdonagh wrote: post #13

I wrote up a bit about it here, from what I could piece together: https://lifeinthesingularity.com/p/googles-breakthroughs-in-...

Nice writing. Reminds me of New Scientist style. (I like NS so that is a compliment). I think the “explain as you go along but be brief style”. Which is nice for getting a feel for the space.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#46

Post by webmaven »

refulgentis wrote: post #40
whimsicalism wrote: post #38

Earlier quoted context omitted.

You would be wrong, but that is fine. Been working with attention since 2018. Why assume I know little and leave snarky comments (and basically a repetition of the prior joke at that, subbing RNN for transformer)?

To playfully invite for you to participate in conversation further, so that I may humbly learn from you. "I don't know what you're talking about" seemed too spartan and austere and aggressive, and you reciprocated politely, if again sparsely, when the other person playfully invited you to elaborate.

Well, you've now made your original intent specific, but in case you didn't draw the requisite lesson I'll make that explicit.

Because text has less bandwidth than almost any other medium, certain forms of humor are much more likely to be understood (in this case, your "gentle playfulness" was taken to be snark, sarcasm, and point scoring).

If you insist on using this and similar forms of humor that, ordinarily, depend quite strongly on intonation to convey intent, you'll have to be much more explicit to avoid being misunderstood. You are going to have actually state your intent explicitly as part of your communication. This need not entirely destroy the humor, for example, you might try something like this:

And so I say to you (playfully, sir, playfully): etc.

Or this:

Yadda yadda yadda. (I kid, I kid!)

The Internet-native forms of this are the humble ;-) or the newer j/k, but I find that it is all too easy to overlook a 3-character sequence, particularly if the passage being so marked is even as long as a single paragraph, but they can serve their purpose when used for the commonplace one-liner.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#47

Post by hackerlight »

maxrumpf wrote: post #36

The abstract and the rest of the paper don't really match imo. It's not really allocating more to some sequences, but just introducing ~dropout. Might be different sides to the same coin, but was still a weird read.

Dropout is at train time this is at inference time. Dropout is random this is determined. Can't compare them.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#48

Post by modeless »

It's a start but it's disappointing that half the layers still have to process every token. It seems like we ought to be able to get to 90% or even 99% savings when these models currently allocate the same compute for outputting "the" as they do for outputting the first digit of the answer of a complicated math problem.

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#49

Post by aiddun »

modeless wrote: post #48

It's a start but it's disappointing that half the layers still have to process every token. It seems like we ought to be able to get to 90% or even 99% savings when these models currently allocate the same compute for outputting "the" as they do for outputting the first digit of the answer of a complicated math problem.

Speculative decoding does this to an extent - using a smaller model to generate its own predictions and putting them in the batch of the bigger model until they diverge

https://huggingface.co/blog/whisper-speculative-decoding

Re: Mixture-of-Depths: Dynamically allocating compute in transformers

#50

Post by brrrrrm »

aiddun wrote: post #49
modeless wrote: post #48

It's a start but it's disappointing that half the layers still have to process every token. It seems like we ought to be able to get to 90% or even 99% savings when these models currently allocate the same compute for outputting "the" as they do for outputting the first digit of the answer of a complicated math problem.

Speculative decoding does this to an extent - using a smaller model to generate its own predictions and putting them in the batch of the bigger model until they diverge https://huggingface.co/blog/whisper-speculative-decoding

It doesn’t. It simply trades compute efficiency by transposing matrix multiplications into “the future.” It doesn’t actually save FLOPs (uses more) and doesn’t work at large batch size
Post reply on HN