Retentive Network: A Successor to Transformer for Large Language Models
1–10 of 22 posts
Re: Retentive Network: A Successor to Transformer for Large Language Models
#2[deleted]
Re: Retentive Network: A Successor to Transformer for Large Language Models
#3This a super interesting paper, but in my oppinion - they do not complete their core claim of offering a generalized mathematical structure for attention/recurrence. The specific structure they propose is very interesting and demonstrably efficient computationally - however they do not show that this approach produces similar accuracy as large LLMs.
I’m anxiously awaiting the follow up where someone tries spending 1MM+ on demonstrating this approaches effectiveness in a large language model context.
Re: Retentive Network: A Successor to Transformer for Large Language Models
#4MIT and Microsoft Researchers Introduce "RetNet" - An 8X Faster Transformer Alternative for AI
Re: Retentive Network: A Successor to Transformer for Large Language Models
#5Amazing - 6.7 billion is significantly larger than I've seen any transformer alternative trained to so far (H3 only went up to 2.7b, e: oops - RWKV goes up to 14b)... cool to see that it appears to be scaling even better and the O(1) & O(N) scaling is great.
Wish there was more consistency on source of training data. Training on just The Pile would enable more clean comparison with most promising transformer alternatives, like H3 and give a better sense of how robust the perplexity improvements cited are.
Re: Retentive Network: A Successor to Transformer for Large Language Models
#6This a super interesting paper, but in my oppinion - they do not complete their core claim of offering a generalized mathematical structure for attention/recurrence. The specific structure they propose is very interesting and demonstrably efficient computationally - however they do not show that this approach produces similar accuracy as large LLMs. I’m anxiously awaiting the follow up where someone tries spending 1M…
> however they do not show that this approach produces similar accuracy as large LLMs.
I think they have demonstrated their case pretty well, unless there is some serious degradation of the scaling - 7b is pretty big.
Re: Retentive Network: A Successor to Transformer for Large Language Models
#7MIT and Microsoft Researchers Introduce "RetNet" - An 8X Faster Transformer Alternative for AI
The claim is parallelism for training which is not fixed speed up, different complexity for inference (constant time), and different complexity for large context inference (linear) - so nothing that can be summarised as 8x - or am I getting this summary wrong?
Re: Retentive Network: A Successor to Transformer for Large Language Models
#8This a super interesting paper, but in my oppinion - they do not complete their core claim of offering a generalized mathematical structure for attention/recurrence. The specific structure they propose is very interesting and demonstrably efficient computationally - however they do not show that this approach produces similar accuracy as large LLMs. I’m anxiously awaiting the follow up where someone tries spending 1M…
> however they do not show that this approach produces similar accuracy as large LLMs. I think they have demonstrated their case pretty well, unless there is some serious degradation of the scaling - 7b is pretty big.
Interestingly, I did see this tweet [0] mentioning a phase shift that occurs in transformers at exactly the scale RetNet stopped at. Probably simply coincidental but I was previously unaware of this phenomenon at such a scale in transformers.
[0] https://twitter.com/gordic_aleksa/status/1682479676910870529
Re: Retentive Network: A Successor to Transformer for Large Language Models
#9Amazing - 6.7 billion is significantly larger than I've seen any transformer alternative trained to so far (H3 only went up to 2.7b, e: oops - RWKV goes up to 14b)... cool to see that it appears to be scaling even better and the O(1) & O(N) scaling is great. Wish there was more consistency on source of training data. Training on just The Pile would enable more clean comparison with most promising transformer alternat…
RWKV has 14B version.
Re: Retentive Network: A Successor to Transformer for Large Language Models
#10The key differences between multi-head attention in Transformers and the proposed multi-scale retention (MSR) in RetNets are:
- Retention replaces the softmax in attention with an exponential decay along the sequence dimension. This allows formulating retention in a recurrent form for efficient O(1) inference.
- Retention heads use different decay rates (gamma values) for multi-scale modeling. Attention heads use the same softmax.
- Retention outputs are normalized per-head with GroupNorm before concatenation. Attention uses LayerNorm on the concatenated output.
- Retention can be computed in parallel, recurrent, or chunkwise recurrent modes. Attention is only parallel.
- The recurrent form enables RetNets to summarize long previous context into a fixed-size state during inference. Attention recomputes on the full context each step.
- So in summary, retention adapts attention to enable recurrent modeling and multi-scale decays. This provides efficiency benefits and competitive performance to Transformers.