Bamba in italian slang is cocaine, just to tell you
Bamba: An open-source LLM that crosses a transformer with an SSM
71–73 of 73 posts
Re: Bamba: An open-source LLM that crosses a transformer with an SSM
#72This type of architecture is definitely the future. Unlimited attn is a dead end. As a human you don't need to scan an entire book just to guess what the next word will be and LLMs shouldn't need that either.
Humans can re-attend to material whenever necessary (i.e you can just re-read a book, re-watch a documentary etc when you feel you have missed crucial context) so it's not the end of the world. These SSMs or modern RNNs can't and if crucial context has been discarded by the end of the query then well too bad. Transformers are of course always re-attending so not an issue for them either. Until that issue is resolved,…
As you said. Transformers are using linear attention for each token. It's just that n times n is quadratic. There is no way around this other than by adding a separate token that indicates rerunning the SSM from the beginning. Then you have a dynamically scaling system that seamlessly switches between linear and quadratic complexity depending on the problem.
MLA is probably the closest thing that is in-between both.
Re: Bamba: An open-source LLM that crosses a transformer with an SSM
#73Earlier quoted context omitted.
How often do you go back an rewatch earlier parts of a movie? I hardly ever do this. In the cinema, theater, or when listening to the radio it’s simply impossible and it still works.
You are mentioning avenues that are largely for entertainment. Sure you might not go back to re-attend for those. If you will be tested or are doing research, are you really looking at a large source once ?
It’s do easy to come up with serious non-entertainment examples, I‘m sure you don’t need my help finding them.