Ring attention with blockwise transformers for near-infinite context
1–10 of 23 posts
Re: Ring attention with blockwise transformers for near-infinite context
#2It's cool to see the founder of a major company still write papers.
Re: Ring attention with blockwise transformers for near-infinite context
#3Rather than all this effort to work around the flaws of the transformer model, maybe researchers should be looking for a better architecture altogether.
The absolutely insane amount of compute that transformers consume could probably be better used for neuroevolutionary search.
Re: Ring attention with blockwise transformers for near-infinite context
#4Rather than all this effort to work around the flaws of the transformer model, maybe researchers should be looking for a better architecture altogether. The absolutely insane amount of compute that transformers consume could probably be better used for neuroevolutionary search.
The unreasonable effectiveness of transformer attention outweighs a lot of the downsides of limited context length for many applications.
Re: Ring attention with blockwise transformers for near-infinite context
#5It's cool to see the founder of a major company still write papers.
[deleted]
Re: Ring attention with blockwise transformers for near-infinite context
#6Rather than all this effort to work around the flaws of the transformer model, maybe researchers should be looking for a better architecture altogether. The absolutely insane amount of compute that transformers consume could probably be better used for neuroevolutionary search.
The unreasonable effectiveness of transformer attention outweighs a lot of the downsides of limited context length for many applications.
I mean ... if you think about it, attention changes the effective weights of a model.
I am fairly certain that if you try, you can show that for any particular sequence of tokens of length N, the N-1 tokens induce a residual FFNN that results in exactly the same distribution over the next tokens given just the Nth.
Re: Ring attention with blockwise transformers for near-infinite context
#7Earlier quoted context omitted.
The unreasonable effectiveness of transformer attention outweighs a lot of the downsides of limited context length for many applications.
I mean ... if you think about it, attention changes the effective weights of a model. I am fairly certain that if you try, you can show that for any particular sequence of tokens of length N, the N-1 tokens induce a residual FFNN that results in exactly the same distribution over the next tokens given just the Nth.
Sounds interesting, try it and share your results here :)
Re: Ring attention with blockwise transformers for near-infinite context
#8I am disappointed to see a paper with the phrase, in the title no less, "Near-Infinite". Something is either infinite or not; there can be no "near".
Re: Ring attention with blockwise transformers for near-infinite context
#9Earlier quoted context omitted.
The unreasonable effectiveness of transformer attention outweighs a lot of the downsides of limited context length for many applications.
I mean ... if you think about it, attention changes the effective weights of a model. I am fairly certain that if you try, you can show that for any particular sequence of tokens of length N, the N-1 tokens induce a residual FFNN that results in exactly the same distribution over the next tokens given just the Nth.
You may be interested in "Linear Transformers Are Secretly Fast Weight Programmers": https://arxiv.org/abs/2102.11174
Re: Ring attention with blockwise transformers for near-infinite context
#10I am disappointed to see a paper with the phrase, in the title no less, "Near-Infinite". Something is either infinite or not; there can be no "near".
[deleted]