51–60 of 89 posts
I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
Most of the original MoE implementations around LLMs were in fact recursive
I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
We can name these hypothetical objects Recursive Neural Networks.
Earlier quoted context omitted.
To playfully invite for you to participate in conversation further, so that I may humbly learn from you. "I don't know what you're talking about" seemed too spartan and austere and aggressive, and you reciprocated politely, if again sparsely, when the other person playfully invited you to elaborate.
Well, you've now made your original intent specific, but in case you didn't draw the requisite lesson I'll make that explicit. Because text has less bandwidth than almost any other medium, certain forms of humor are much more likely to be understood (in this case, your "gentle playfulness" was taken to be snark, sarcasm, and point scoring). If you insist on using this and similar forms of humor that, ordinarily, depe…
Earlier quoted context omitted.
To playfully invite for you to participate in conversation further, so that I may humbly learn from you. "I don't know what you're talking about" seemed too spartan and austere and aggressive, and you reciprocated politely, if again sparsely, when the other person playfully invited you to elaborate.
Well, you've now made your original intent specific, but in case you didn't draw the requisite lesson I'll make that explicit. Because text has less bandwidth than almost any other medium, certain forms of humor are much more likely to be understood (in this case, your "gentle playfulness" was taken to be snark, sarcasm, and point scoring). If you insist on using this and similar forms of humor that, ordinarily, depe…
Simplified Intro Version: Imagine you have a smart assistant that can understand and process the words you say to it. Usually, this assistant pays equal attention to every word you say, no matter how important or unimportant each word is to the overall meaning of your message. Now, imagine that we found a way to teach the assistant to be smarter about how it uses its "brain power." Instead of giving equal attention t…
The idea that we want models not to have to use the same amount of compute for every token has been around for a while. This is the first compelling mechanism I've seen for doing it.
> Equipped with these new methods, we can sample autoregressively by choosing to route tokens to or around a block based on the router’s output, which does not depend on any information from future tokens. We provide empirical evidence that this is a relatively easy auxiliary task that quickly achieves 99% accuracy.
Does anyone else find this is a bit surprising?
Earlier quoted context omitted.
To playfully invite for you to participate in conversation further, so that I may humbly learn from you. "I don't know what you're talking about" seemed too spartan and austere and aggressive, and you reciprocated politely, if again sparsely, when the other person playfully invited you to elaborate.
Well, you've now made your original intent specific, but in case you didn't draw the requisite lesson I'll make that explicit. Because text has less bandwidth than almost any other medium, certain forms of humor are much more likely to be understood (in this case, your "gentle playfulness" was taken to be snark, sarcasm, and point scoring). If you insist on using this and similar forms of humor that, ordinarily, depe…
Simplified Intro Version: Imagine you have a smart assistant that can understand and process the words you say to it. Usually, this assistant pays equal attention to every word you say, no matter how important or unimportant each word is to the overall meaning of your message. Now, imagine that we found a way to teach the assistant to be smarter about how it uses its "brain power." Instead of giving equal attention t…
I understand this is ELI5, but doesn’t attention already do this, in the way you described? It pays specific focus to the most contextual words in the prior sequence.
The innovation here is to prioritize tokens so that some tokens have more or less processor time.