Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

Post by fheinsen » Wed, Feb 04, 2026, 6:47 PM UTC

riemannzeta wrote: post #42

Neat result. The symmetry exploitation here reminds me of recent work connecting neural network training dynamics to renormalization group theory. Charles Martin's SETOL paper https://arxiv.org/abs/2507.17912 shows that well-trained layers converge to something like an RG fixed point—the eigenvalue spectrum of the weight matrix develops power-law tails with exponent α ≈ 2, which is the signature of scale invariance.…

Post by orlp » Wed, Feb 04, 2026, 6:59 PM UTC

logicchains wrote: post #33

kristjansson wrote: post #26

Earlier quoted context omitted.

> self-attention is efficiently computable to arbitrary precision with constant cost per token This paper at least aspires to reproduce 'true' attention, which distinguishes it from many of the others. TBD if its successful in that.

It can't be successful at that any more than 1+1 can equal 3. Fundamentally, if every token wants to be able to look at every previous token without loss of information, it must be O(n^2); N tokens looking at N tokens is quadratic. Any sub-quadratic attention must hence necessarily lose some information and be unable to support perfect recall on longer sequences.

    total = 0
    for i in range(len(a)):
        for j in range(len(b)):
            total += a[i] * b[j]

Post by kristjansson » Wed, Feb 04, 2026, 7:06 PM UTC

jcarreiro wrote: post #31

thomasahle wrote: post #9

There's a graveyard of 100s of papers with "approximate near linear time attention." They always hope the speed increase makes up for the lower quality, but it never does. The quadratic time seems inherent to the problem. Indeed, there are lower bounds showing that sub n^2 algorithms can't work: https://arxiv.org/pdf/2302.13214

The paper says that: > In practice, we find that four Taylor terms (P = 4) suffice for recovering conventional attention with elementwise errors of approximately the same magnitude as Float16 resolution, acceptable for many AI applications. ie., the claim is that this method reproduces the results of conventional attention, up to float16 numerical precision.

Post by mapontosevenths » Wed, Feb 04, 2026, 7:30 PM UTC

slashdave wrote: post #58

mapontosevenths wrote: post #5

This uses the Taylor approximation to approximate softmax, but that IS only an approximation. I wonder exactly how much that trade-off costs in terms of accuracy vs performance? I note that they say it's close to float16 with four Taylor terms. My other concern would be that Taylor itself is fairly complex. I wonder how well GPU's handle this in comparison to good old fashioned softmax? The last time I used Taylor wi…

If the model learns by using the approximate softmax, then why does it matter? We only need the behavior of softmax, not an exact numerical solution.

Post by ehsanu1 » Wed, Feb 04, 2026, 8:06 PM UTC

vlovich123 wrote: post #29

seanhunter wrote: post #14

Earlier quoted context omitted.

I read that too, but I wondered whether elementwise error is the right metric. Surely the actual error metric should be to evaluate model performance for a conventional transformer model and then the same model with the attention mechanism replaced by this 4th order Taylor approximation?

Bounded error weights by definition is a more strict evaluation criterion than “performance” metrics through running the model.