Viewing profile — fheinsen

The github repository's first toy example is with 8 Taylor terms, applied to a context with 1B tokens, each token with 1K heads: https://github.com/glassroom/sata_attention That to…

Unlike previous efforts, which typically stop at a low-order (e.g., quadratic) term of the Taylor expansion, this work derives a succinct, efficient, parallel general method for ap…

As far as I know, there is no widely accepted explanation for context rot. Numerical error in long sequences of query-key dot-products may be a key factor.

Yes, there must be a connection. While adaptive truncation may prove impractical, it should be possible to measure spectral statistics on sample data, and specify a different fixed…

[3] is linear attention, https://arxiv.org/abs/2006.16236 , a well-known result with ~3K citations: https://scholar.google.com/scholar_lookup?arxiv_id=2006.1623...

This is a form of linear attention ( https://arxiv.org/abs/2006.16236 ) that approximates standard scaled dot-product attention to arbitrary precision, by adding Taylor terms in an…

The method is more general. The github repository's first example is with eight Taylor terms (P = 8).

As the error via linear approximation approaches similar magnitude as numerical error via quadratic computation, don’t the two start becoming comparable in practice ? I ask because…

Yes. See this comment for context: https://news.ycombinator.com/item?id=45611863

To the best of our knowledge, this is the first time anyone has successfully trained a non-diagonal RNN computed in parallel, via prefix scan, without requiring any form of stabili…

The manuscript formally defines GOOMs as a set of mathematical objects, shows that floating-point formats are a special case of GOOMs, and notes that they extend prior work on loga…

The paper formally defines GOOMs as the subset of the complex plane that elementwise exponentiates to the real line. We'll update the README to make that clearer. Thank you for poi…

> GOOMs generalize the concept of "order of magnitude" to incorporate complex numbers that exponentiate to real ones. As with ordinary orders of magnitude, GOOMs are more stable th…

repo: https://github.com/glassroom/generalized_orders_of_magnitude

I'm the author of that paper. Happy to answer questions about it here.

Hi HN, I'm the author of this paper. As it turns out, I posted it and answered a few questions about it on HN a few days ago, on this thread: https://news.ycombinator.com/item?id=2…

Thanks. Will do (work permitting!). FWIW, a while back I reimplemented and tinkered a bit with the Sabour et al. version too... and did not see much promise in it. Note that the ro…

FYI, I reached out to two of those individuals (one is a CVPR reviewer, it turns out) and both suggested I first upload this to arXiv, so I did that yesterday. The paper is now sta…