The Annotated S4 • Hacker News Board

Re: The Annotated S4

#11

Post by efm » Mon, Feb 12, 2024, 9:51 PM UTC

ptojr wrote: post #10

imjonse wrote: post #2

A lot of intimidating math that will make all self-attention tutorials seem like a walk in the park in comparison. Luckily subsequent state space models building on S4 (DSS, S4D and newer ones like Mamba) simplified the primitives and the math used.

Can someone point me to DSS and S4D papers?

DSS: Diagonal State Spaces are as Effective as Structured State Spaces (https://arxiv.org/abs/2203.14343)

S4D: On the Parameterization and Initialization of Diagonal State Space Models (https://arxiv.org/abs/2206.11893)

Re: The Annotated S4

#12

Post by ssivark » Mon, Feb 12, 2024, 10:56 PM UTC

imjonse wrote: post #2

A lot of intimidating math that will make all self-attention tutorials seem like a walk in the park in comparison. Luckily subsequent state space models building on S4 (DSS, S4D and newer ones like Mamba) simplified the primitives and the math used.

Well, but this stuff is also much more principled and much better understood (by construction) than why/how a transformer works. The price of actual understanding, and being able to make precise statements, is that the statements will be precise and detailed (ie likely involve math).

Re: The Annotated S4

#13

Post by ptojr » Tue, Feb 13, 2024, 1:50 AM UTC

efm wrote: post #11

ptojr wrote: post #10

Earlier quoted context omitted.

Can someone point me to DSS and S4D papers?

DSS: Diagonal State Spaces are as Effective as Structured State Spaces ( https://arxiv.org/abs/2203.14343 ) S4D: On the Parameterization and Initialization of Diagonal State Space Models ( https://arxiv.org/abs/2206.11893 )

Thank you!