Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation • Hacker News Board

Re: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

#81

Post by direwolf20 » Wed, Feb 04, 2026, 11:45 PM UTC

logicchains wrote: post #74

hellohello2 wrote: post #44

Earlier quoted context omitted.

I'm not saying if the paper is correct or not (since I can't tell), but I don't think your argument really holds. Consider applying it to multiplication: Fundamentally, multiplication need to look at every pair of integer from the two input numbers. It must be O(n^2); N digits looking at N other digits is quadratic. Any sub-quadratic multiplication must hence necessarily lose some information.

Multiplication has some properties like being cumulative. If we assume the sequence has any specific properties then we no longer have a general sequence model.

I think you meant commutative.

Attention also has some specific properties.

And sometimes results are just unexpected. Did you know that anything a Turing machine can do in t tome steps, a different Turing machine can do in O(sqrt(t log t)) memory cells? https://news.ycombinator.com/item?id=44055347

Re: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

#82

Post by direwolf20 » Wed, Feb 04, 2026, 11:52 PM UTC

logicchains wrote: post #73

naasking wrote: post #60

Earlier quoted context omitted.

Your argument just assumes there is no latent structure that can be exploited. That's a big assumption.

It's a necessary assumption for the universal approximation property; if you assume some structure then your LLM can no longer solve problems that don't fit into that structure as effectively.

Neural nets are structured as matrix multiplication, yet, they are universal approximators.

Re: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

#83

Post by direwolf20 » Wed, Feb 04, 2026, 11:54 PM UTC

dave_universetf wrote: post #59

SubiculumCode wrote: post #30

Earlier quoted context omitted.

Dumb question: Can inference be done in a reverse pass? Outputs predicting inputs?

Strictly speaking: no. The "forward pass" terminology does not imply that there exists a "reverse pass" that does the same kind of computation. Rather, it's describing two different kinds of computation, and the direction they occur in. The forward pass is propagating from inputs to outputs, computing the thing the model was trained for. The reverse/backwards pass is propagating from outputs back to inputs, but it's…

I do want to see ChatGPT running upwards on my screen now, predicting earlier and earlier words in a futile attempt to explain a nonsense conclusion. We could call it ChatJeopardy.

Re: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

#84

Post by noosphr » Thu, Feb 05, 2026, 12:02 AM UTC

orlp wrote: post #63

logicchains wrote: post #33

Earlier quoted context omitted.

It can't be successful at that any more than 1+1 can equal 3. Fundamentally, if every token wants to be able to look at every previous token without loss of information, it must be O(n^2); N tokens looking at N tokens is quadratic. Any sub-quadratic attention must hence necessarily lose some information and be unable to support perfect recall on longer sequences.

> N tokens looking at N tokens is quadratic Convolving two arrays can be done perfectly accurately in O(n log n), despite every element being combined with every other element. Or consider the even more basic sum of products a[i] * b[j] for all possible i, j: total = 0 for i in range(len(a)): for j in range(len(b)): total += a[i] * b[j] This can be computed in linear time as sum(a) * sum(b). Your logic that 'the resu…

Convolution is a local operation.

Attention is a global operation.

Re: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

#85

Post by noosphr » Thu, Feb 05, 2026, 12:05 AM UTC

direwolf20 wrote: post #82

logicchains wrote: post #73

Earlier quoted context omitted.

It's a necessary assumption for the universal approximation property; if you assume some structure then your LLM can no longer solve problems that don't fit into that structure as effectively.

Neural nets are structured as matrix multiplication, yet, they are universal approximators.

You're missing the non-linear activations.

Re: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

#86

Post by torginus » Thu, Feb 05, 2026, 12:45 AM UTC

fheinsen wrote: post #40

jcarreiro wrote: post #31

Earlier quoted context omitted.

The paper says that: > In practice, we find that four Taylor terms (P = 4) suffice for recovering conventional attention with elementwise errors of approximately the same magnitude as Float16 resolution, acceptable for many AI applications. ie., the claim is that this method reproduces the results of conventional attention, up to float16 numerical precision.

The method is more general. The github repository's first example is with eight Taylor terms (P = 8).

I'm clueless about this whole thing, but from my EE education I remember that in general:

Taylor approximations converge slowly in terms of error if the function they're representing is discontinuous (the error disappears quadratically if continuous, linearly if not), and they tend to create highly energetic swings near discontinuties (similarly to Fourier series with Gibbs oscillations).

Moreover, Taylor series are inherently nonlinear, and much of the mathematical toolset around AI assumes general linearity (cue linear algebra), with the exception of sigmoids , and going beyond cubic approximations tends to make errors worse (as expressed in SNR).

Re: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

#87

Post by [deleted] » Thu, Feb 05, 2026, 12:45 AM UTC

noosphr wrote: post #84

orlp wrote: post #63

Earlier quoted context omitted.

> N tokens looking at N tokens is quadratic Convolving two arrays can be done perfectly accurately in O(n log n), despite every element being combined with every other element. Or consider the even more basic sum of products a[i] * b[j] for all possible i, j: total = 0 for i in range(len(a)): for j in range(len(b)): total += a[i] * b[j] This can be computed in linear time as sum(a) * sum(b). Your logic that 'the resu…

Convolution is a local operation. Attention is a global operation.

[deleted]

Re: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

#88

Post by cptroot » Thu, Feb 05, 2026, 12:50 AM UTC

kristjansson wrote: post #65

jcarreiro wrote: post #31

Earlier quoted context omitted.

The paper says that: > In practice, we find that four Taylor terms (P = 4) suffice for recovering conventional attention with elementwise errors of approximately the same magnitude as Float16 resolution, acceptable for many AI applications. ie., the claim is that this method reproduces the results of conventional attention, up to float16 numerical precision.

> approximately the same magnitude and they really do mean that, their results show +/- 1 on log10 plots.

I don't think this is an accurate characterization of the error magnitude? Their error plots (from appendix 3) are all showing `log_10(|Y - \dot{Y}|)` as having a median of ~-3 (difference of 0.001) and a max of ~1.5 (difference of 0.035), and this is with only 3 Taylor terms.

Re: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

#89

Post by nine_k » Thu, Feb 05, 2026, 2:02 AM UTC

hellohello2 wrote: post #44

logicchains wrote: post #33

Earlier quoted context omitted.

It can't be successful at that any more than 1+1 can equal 3. Fundamentally, if every token wants to be able to look at every previous token without loss of information, it must be O(n^2); N tokens looking at N tokens is quadratic. Any sub-quadratic attention must hence necessarily lose some information and be unable to support perfect recall on longer sequences.

I'm not saying if the paper is correct or not (since I can't tell), but I don't think your argument really holds. Consider applying it to multiplication: Fundamentally, multiplication need to look at every pair of integer from the two input numbers. It must be O(n^2); N digits looking at N other digits is quadratic. Any sub-quadratic multiplication must hence necessarily lose some information.

Integer multiplication x * y can be trivially done in O(k): k = log₂(min(x, y)). This is because we can do addition in constant time, adding all bits in parallel.

By combining many more adding units, we can do (fixed-size) multiplication in constant time, too: https://en.wikipedia.org/wiki/Dadda_multiplier

Re: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

#90

Post by charleshmartin » Thu, Feb 05, 2026, 2:14 AM UTC

riemannzeta wrote: post #42

Neat result. The symmetry exploitation here reminds me of recent work connecting neural network training dynamics to renormalization group theory. Charles Martin's SETOL paper https://arxiv.org/abs/2507.17912 shows that well-trained layers converge to something like an RG fixed point—the eigenvalue spectrum of the weight matrix develops power-law tails with exponent α ≈ 2, which is the signature of scale invariance.…

Right. If the dynamics of training are governed by RG flow, then the best optimization path should remove redundant directions, as specified by the RG operator(s)