Mamba outperforms transformers "everywhere we tried" • Hacker News Board

Re: Mamba outperforms transformers "everywhere we tried"

#21

Post by psb217 » Tue, Dec 12, 2023, 3:13 AM UTC

dontreact wrote: post #5

If you want to really dive into the math and motivation for this entirely new class of models (state space models) I highly recommend reading Albert Gu’s thesis. https://searchworks.stanford.edu/view/14784021 I tried to read the Mamba paper and I was lost but after reading his thesis it was a lot to understand it. The part I’m still struggling with is the section on how Mamba trains so efficiently, it’s surprising si…

Due to the particular form of the recurrent update of the hidden state, there's actually a parallel algorithm for computing the recurrence over length N in log(N) time via dynamic programming. Note, you don't save FLOPs, you just save "sequential depth" in the computation through clever parallelization. It's sort of an extension of the fast parallel scan described here: https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-co....

Re: Mamba outperforms transformers "everywhere we tried"

#22

Post by dang » Tue, Dec 12, 2023, 3:50 AM UTC

squigz wrote: post #19

This is a reddit post which links to a Twitter post [0] which links to this paper [1] and this repo [2] [0] - https://nitter.net/_albertgu/status/1731727672286294400 [1] - https://arxiv.org/abs/2312.00752 [2] - https://github.com/state-spaces/mamba Out of curiosity, does anyone feel as though there's any benefit to linking to reddit when we can link to whatever the reddit post actually is? I for one do not click the…

Ah yes, plus your link [1] already had significant attention here (thanks!):

Mamba: Linear-Time Sequence Modeling with Selective State Spaces - https://news.ycombinator.com/item?id=38522428 - Dec 2023 (34 comments)

> does anyone feel as though there's any benefit to linking to reddit

You were certainly right in this case, but I'm not ready to ban reddit.com because I think there are still some positive threads that come in through that window. If anyone disagrees strongly enough to peruse https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... and make the case against, the court will grant standing :)

Re: Mamba outperforms transformers "everywhere we tried"

#23

Post by wenc » Tue, Dec 12, 2023, 4:10 AM UTC

malwrar wrote: post #15

Question for anyone reading the paper; what exactly is the notation in these two stages below supposed to mean? 𝑦(𝑡) = Ch(𝑡) (1b) 𝑦𝑡 = Ch𝑡 (2b) This is from the section describing state space models. The paragraphs before seem to describe e.g. 𝑦(𝑡) as being presumably the real value element in function/sequence 𝑦 evaluated at time 𝑡. However, it then randomly adds the 𝑦𝑡 (note the 𝑡 is supposed to be sub…

As the other comment suggested, it's the discrete subscript. Normally t is notation for continuous time, and k the discrete time subscript, but all this is just convention.

yₖ = Chₖ

Re: Mamba outperforms transformers "everywhere we tried"

#24

Post by malwrar » Tue, Dec 12, 2023, 5:26 AM UTC

molticrystal wrote: post #18

malwrar wrote: post #15

Question for anyone reading the paper; what exactly is the notation in these two stages below supposed to mean? 𝑦(𝑡) = Ch(𝑡) (1b) 𝑦𝑡 = Ch𝑡 (2b) This is from the section describing state space models. The paragraphs before seem to describe e.g. 𝑦(𝑡) as being presumably the real value element in function/sequence 𝑦 evaluated at time 𝑡. However, it then randomly adds the 𝑦𝑡 (note the 𝑡 is supposed to be sub…

y(t) is likely continuous and y_t , the subscript form, is likely the discrete version. For example it is common in digital devices to sample something continuous(analog) every unit of time and use whatever value was captured at that moment for whatever the goal is, ideally if the something is sampled frequently enough you can even recreate the continuous version from the values.

Thanks for clarifying that! I think this makes sense, I wish I had a stronger math background so that this kind of thing doesn’t trip me up as much.

Re: Mamba outperforms transformers "everywhere we tried"

#25

Post by loxias » Tue, Dec 12, 2023, 5:54 AM UTC

squigz wrote: post #19

This is a reddit post which links to a Twitter post [0] which links to this paper [1] and this repo [2] [0] - https://nitter.net/_albertgu/status/1731727672286294400 [1] - https://arxiv.org/abs/2312.00752 [2] - https://github.com/state-spaces/mamba Out of curiosity, does anyone feel as though there's any benefit to linking to reddit when we can link to whatever the reddit post actually is? I for one do not click the…

Speaking purely for myself, (curious what other people think) my preference is _always_ for the link to go to the most "principal" source. Then, in the comments, add a link to a pop news article or the comments section.

I hate links to twitter posts, feels like undeserved promotion of Elon Musk. Just find what the twitter links to, or the actual research paper, and submit that.

Links to reddit are okay, it's devolving, for sure, but depending on topic sometimes I find more intelligent comments there. (RARELY)

Re: Mamba outperforms transformers "everywhere we tried"

#26

Post by behnamoh » Tue, Dec 12, 2023, 10:36 PM UTC

loxias wrote: post #25

squigz wrote: post #19

This is a reddit post which links to a Twitter post [0] which links to this paper [1] and this repo [2] [0] - https://nitter.net/_albertgu/status/1731727672286294400 [1] - https://arxiv.org/abs/2312.00752 [2] - https://github.com/state-spaces/mamba Out of curiosity, does anyone feel as though there's any benefit to linking to reddit when we can link to whatever the reddit post actually is? I for one do not click the…

Speaking purely for myself, (curious what other people think) my preference is _always_ for the link to go to the most "principal" source. Then, in the comments, add a link to a pop news article or the comments section. I hate links to twitter posts, feels like undeserved promotion of Elon Musk. Just find what the twitter links to, or the actual research paper, and submit that. Links to reddit are okay, it's devolvin…

> I hate links to twitter posts, feels like undeserved promotion of Elon Musk. Just find what the twitter links to, or the actual research paper, and submit that.

Treat knowledge with respect regardless of where you get it from.

Re: Mamba outperforms transformers "everywhere we tried"

#27

Post by mcint » Wed, Dec 13, 2023, 9:09 PM UTC

bananaheel wrote: post #8

Still chewing but my takeaways thus far: SSM models strengths are with continuous data like audio and video, they struggle with discrete data like text/ DNA. This newest architecture uses selective attention to try to address the weaknesses around discrete data with some loss in performance in continuous tasks, empirically shown here with audio. The empirical exploration was limited to smaller size models, the perfor…

The distinction I remember from the paper, is that discrete data (text, DNA) could be modeled effectively with only real-valued components, while continuous data (audio, video) benefited from complex numbers -- in their state. Evidently summarized from the authors statements characterizing existing work, preexisting wisdom on SSM/S4 models.

https://youtubetranscript.com/?v=ouF-H35atOY (same video, I'd watched previous to seeing this hn post)

  other model details the authors note
  that most prior State space models use
  complex numbers in their state but it
  has been empirically observed that
  completely real valued State space
  models seem to work fine and possibly
  even better in some settings so they use
  real values as the default which work
  well for all but one of their tasks next

just following this, another impressive snippet

  it succeeds on test sequence lengths of
  up to a million tokens which is 4,000
  times longer than it saw during training
  while none of the other methods compared
  to generalize to Beyond twice their
  training length