31–40 of 89 posts
Earlier quoted context omitted.
i know you're jesting but RNNs are recursive along the sequence length where I am describing recursion along the depth.
Like decode the next token, then adjust what you're paying attention to, then decode it again?
Earlier quoted context omitted.
How is attention basically routing?
Think of it like an edge flow matrix
Imagine you have a smart assistant that can understand and process the words you say to it. Usually, this assistant pays equal attention to every word you say, no matter how important or unimportant each word is to the overall meaning of your message.
Now, imagine that we found a way to teach the assistant to be smarter about how it uses its "brain power." Instead of giving equal attention to every word, the assistant learns to focus more on the words that are most important for understanding what you mean. It can even adjust this focus on the fly, paying more attention to different words depending on the context of your message.
To make sure the assistant doesn't get overwhelmed, we also set a limit on how much total "brain power" it can use at any given time. It's like giving the assistant a budget and saying, "You can only spend your brain power on a certain number of words at a time." The assistant then has to decide which words are most important to focus on.
Even with this limit, the assistant is still flexible in how it uses its brain power. It might spend more on certain words and less on others, depending on what you're saying. This means that while we always know the total amount of brain power the assistant is using, it can adapt to different situations and prioritize what's most important.
When we teach the assistant using this method, it not only learns to focus its attention intelligently but also does so very efficiently. It can understand you just as well as an assistant that pays equal attention to every word, but it uses less brain power overall. This makes the assistant much faster at responding to you and processing new information.
Earlier quoted context omitted.
Like decode the next token, then adjust what you're paying attention to, then decode it again?
Isn't it the only way to, say,understand a pun?
The abstract and the rest of the paper don't really match imo. It's not really allocating more to some sequences, but just introducing ~dropout. Might be different sides to the same coin, but was still a weird read.
That said, it's a pretty simple change to make the approach work in the way you describe (allocating more to some sequences and not others) by changing the group across which the top-k works. In the paper we use the time (sequence) dimension, but one could also use the batch * time dimension, which would result in asymmetric allocation across sequences
Earlier quoted context omitted.
Isn't it the only way to, say,understand a pun?
That is exactly how LLM inference is performed, so I'm being cheeky (I'm 99% sure anyone proposing anything in this thread is someone handwaving based on limited understanding)
Why assume I know little and leave snarky comments (and basically a repetition of the prior joke at that, subbing RNN for transformer)?
Neat!
Earlier quoted context omitted.
That is exactly how LLM inference is performed, so I'm being cheeky (I'm 99% sure anyone proposing anything in this thread is someone handwaving based on limited understanding)
You would be wrong, but that is fine. Been working with attention since 2018. Why assume I know little and leave snarky comments (and basically a repetition of the prior joke at that, subbing RNN for transformer)?