11–20 of 27 posts
Title is misleading. “Everywhere we tried” does not include actually beating the performance (or getting particularly close to) sota transformers of the same size. That said, these models were trained for less time than the sota models so there is promise.
Still chewing but my takeaways thus far: SSM models strengths are with continuous data like audio and video, they struggle with discrete data like text/ DNA. This newest architecture uses selective attention to try to address the weaknesses around discrete data with some loss in performance in continuous tasks, empirically shown here with audio. The empirical exploration was limited to smaller size models, the perfor…
Title is misleading. “Everywhere we tried” does not include actually beating the performance (or getting particularly close to) sota transformers of the same size. That said, these models were trained for less time than the sota models so there is promise.
This is the reddit title almost verbatim. If you have a problem with that, you better leave a comment in that thread.
Earlier quoted context omitted.
This is the reddit title almost verbatim. If you have a problem with that, you better leave a comment in that thread.
“please use the original title, unless it is misleading or linkbait” -- https://news.ycombinator.com/newsguidelines.html
𝑦(𝑡) = Ch(𝑡) (1b) 𝑦𝑡 = Ch𝑡 (2b)
Title is misleading. “Everywhere we tried” does not include actually beating the performance (or getting particularly close to) sota transformers of the same size. That said, these models were trained for less time than the sota models so there is promise.
Still chewing but my takeaways thus far: SSM models strengths are with continuous data like audio and video, they struggle with discrete data like text/ DNA. This newest architecture uses selective attention to try to address the weaknesses around discrete data with some loss in performance in continuous tasks, empirically shown here with audio. The empirical exploration was limited to smaller size models, the perfor…
[flagged]
Question for anyone reading the paper; what exactly is the notation in these two stages below supposed to mean? 𝑦(𝑡) = Ch(𝑡) (1b) 𝑦𝑡 = Ch𝑡 (2b) This is from the section describing state space models. The paragraphs before seem to describe e.g. 𝑦(𝑡) as being presumably the real value element in function/sequence 𝑦 evaluated at time 𝑡. However, it then randomly adds the 𝑦𝑡 (note the 𝑡 is supposed to be sub…
For example it is common in digital devices to sample something continuous(analog) every unit of time and use whatever value was captured at that moment for whatever the goal is, ideally if the something is sampled frequently enough you can even recreate the continuous version from the values.
[0] - https://nitter.net/_albertgu/status/1731727672286294400
[1] - https://arxiv.org/abs/2312.00752
[2] - https://github.com/state-spaces/mamba
Out of curiosity, does anyone feel as though there's any benefit to linking to reddit when we can link to whatever the reddit post actually is? I for one do not click the link and read discussion on reddit - if I wanted that sort of discussion, I would browse there, not HN.
This is a reddit post which links to a Twitter post [0] which links to this paper [1] and this repo [2] [0] - https://nitter.net/_albertgu/status/1731727672286294400 [1] - https://arxiv.org/abs/2312.00752 [2] - https://github.com/state-spaces/mamba Out of curiosity, does anyone feel as though there's any benefit to linking to reddit when we can link to whatever the reddit post actually is? I for one do not click the…