Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba Explained

thegradient.pub

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Mamba Explained

#31

Post by derbOac »

jongjong wrote: post #27

I find it difficult to understand certain math and science papers/articles due to ambiguous use of language. For example "all previous tokens can be passed to the current token." That seems like a poorly constructed sentence. A token is not a function and it's not an algorithm either... How can you pass tokens to a token? This type of ambiguous language in academic papers makes it hard to read... Maybe the phrase 'ev…

I agree although I've always interpreted it as a combination of difficulty explaining complex architecture, and also not really understanding why things work the way they do. A lot of modern AI sits in this kind of quasi-empirical realm just above (in an emergent properties sense) analytic math and statistics, and it seems like there's not a very good integrative account or understanding of what's going on, or a way of deriving what direction to go in. So you end up with poor explanations in part because the authors of the structures themselves don't quite understand why things are working as they are.

Re: Mamba Explained

#32

Post by anon291 »

jongjong wrote: post #29
king_magic wrote: post #28

Earlier quoted context omitted.

that's not what it says in the article. it actually says "information from all previous tokens can be passed to the current token". that statement is meaningfully different from "all previous tokens can be passed to the current token". and both really makes sense if you understand attention mechanisms.

Sorry for the misquote but it's a distraction from my issue which was with the usage of the word 'passed'. Do you pass information from other tokens to a token in the sense that each token processes information from other tokens? A token isn't a processing unit AFAIK, it's just a word part. The processing is not the responsibility of the token itself. My understanding is that tokens may be associated with each other…

Well they gave the equations so follow closely where the token representations end up and how they're acted upon.

Re: Mamba Explained

#33

Post by jazzyjackson »

anon291 wrote: post #30
andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

Yes transformers are obviously more capable than humans in my opinion. Claude can ingest dozens of pages in seconds and -- in a single shot -- write a summary bringing in relevant passages. The innovation is not the speed, but the lack of recursion or iteration. Humans, even accomplished ones, have to reread sections and really 'internalize' ideas before being able to summarize and very few humans can -- in a single…

> we start making a sentence and then decide halfway through its not going where we like

I'll just add the observation that when we do this it's largely based on feedback receive from the recipient (well, so long as you're talking-with as opposed to talking-at) - we're paying attention to how the audience is paying attention or not, any small facial tics that might betray skepticism or agreement and so on. I'm looking forward to interacting with an LLM that pairs an emotion-vector along with each token it has previously produced.

hume.ai goes a long way analyzing audio, just a matter of time before they're ingesting realtime facial cues to also incorporate their audience's reaction in their choice of what to say next

Re: Mamba Explained

#34

Post by jazzyjackson »

timschmidt wrote: post #20
aCoreyJ wrote: post #11

Earlier quoted context omitted.

We're running out of the ability to make transistors smaller and closer together so beyond some major breakthrough I wouldnt expect Moore's law to continue nearly long enough to get to the point of running GPT4 on consumer hardware in the short term

Ah, but we've just begun stacking transistors in the third dimension.

It was never a solution, Moore's law has more than one dimension as well, not just density but heat dissipation. Can't cool down a transistor that's surrounded by transistors on all sides.

Re: Mamba Explained

#35

Post by rdedev »

koayon wrote: post #23
andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

This is a very fair point! If we had infinite compute then it's undeniable that transformers (i.e. full attention) would be better (exactly as you characterise it) But that's the efficiency-effectiveness tradeoff that we have to make: given that compute is limited, would we prefer attention over shorter sequences or SSMs over longer sequences? The answer is probably "well, it depends on your use case" - I can definit…

If I remember it right, the llm big bird had something like this. For a particular word it would attend strongly with its closer neighbours but weakly to words far from it. Look for sparse attention. I think that's the relevant terminology. Not sure if it matches exactly what you described

Re: Mamba Explained

#36

Post by rdedev »

anon291 wrote: post #30
andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

Yes transformers are obviously more capable than humans in my opinion. Claude can ingest dozens of pages in seconds and -- in a single shot -- write a summary bringing in relevant passages. The innovation is not the speed, but the lack of recursion or iteration. Humans, even accomplished ones, have to reread sections and really 'internalize' ideas before being able to summarize and very few humans can -- in a single…

I view transformers as like the language center of the brain. When we write or speak, especially when it's critical to get things right, we have this ability to think "that doesn't make sense" and start over. I view this recursion as more of a strength than weakness. You can get an LLM to generate an answer and when asked about the validity of the answer it would acknowledge that it got it wrong. This begs the question that if it had perfect recall and understanding why did it give the wrong answer in the first place?

I don't know how the reasoning part comes to us but if we could implant that capability to a transformer model then it would end up pretty good.

Re: Mamba Explained

#37

Post by fisian »

password4321 wrote: post #6

Links to more about Mamba (selective state space models) on HN yesterday: https://news.ycombinator.com/item?id=39853958#39855430

This submission has the same content as the link here (submitted to HN about a month ago):

https://news.ycombinator.com/item?id=39501982 https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html

Re: Mamba Explained

#38

Post by tempaccount420 »

etbebl wrote: post #13

Anyone else keep seeing articles about Mamba and thinking it's about Python/Conda? It's annoying when the new cool thing picks the same name as something else you like that deserves attention.

Sounds like you need a language model to help you categorize Mamba articles into Python and non-Python articles?

Re: Mamba Explained

#39

Post by logicchains »

andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

>Lately I've been wondering... is this a problem, or a strength?

It's a strength; fundamentally it's impossible to achieve the same degree of accuracy with a sub-quadratic attention mechanism: https://arxiv.org/abs/2209.04881 (unless the Strong Exponential Time Hypothesis is false, which is very unlikely, like P=NP).

Re: Mamba Explained

#40

Post by metadat »

andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

What's an SSM?

For the uninitiated (like me), apparently it stands for State Space Models.

Post reply on HN