Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The FFT Strikes Back: An Efficient Alternative to Self-Attention

arxiv.org

Post reply on HN
Print view View original

121–130 of 175 posts

Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention

#121

Post by quantadev »

Since I believe consciousness itself is made of EMF waves, generated by neural activity (rather than synaptic firings themselves, which I view merely as signal carriers like the I/O to/from brains), I'm glad to see it any time FFTs are used in any way in NNs or AI research.

I started to develop my own custom type of MLP (multilayer perceptron), that was going to use frequencies and phase angles (FFT) as the "model weights", but then I decided probably it would only outperform the standard MLP if the training data itself was periodic in nature, rather than with language tokens or even image data. Not sure if that's correct or not since Fourier Series shows us ANY arbitrary function can be simulated via a superposition of waves.

I still believe if we do achieve something amazing (i.e. competitive with SOTA AI models) with a wave-based NN, it won't create any 'true' qualia however, because simulating EMF waves in a computer is not the same as real EMF waves existing. I think even a 100% perfect simulation of a brain in a computer, for example, will always be a 'zombie' (no qualia). This is obvious if consciousness is indeed made of waves; but it's astounding how few NN-researchers seem to be so illiterate in the field of neuroscience that they don't realize how much evidence there is that consciousness is a wave phenomena.

Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention

#123

Post by yobbo »

pointlessone wrote: post #14
jampekka wrote: post #6

Earlier quoted context omitted.

The actual mechanism at least is quite simple. Essentially it takes the FFT of the input embeddings, multiplies it elementwise with weights that are gotten from the input embeddings using an MLP (plus a constant (but learnable) bias) and then runs it through an activation function and finally takes the inverse FFT. The "frequencies" are probably something quite abstract. FFT is often used in ways where there aren't r…

I’m still confused. Does it treat the input tokens as a sampled waveform? I mean, say I have some text file in ASCII. Do I then just pretend it’s raw wav and do FFT on it? I guess it can give me some useful information (like does it look like any particular natural language or is it just random; sometimes used in encrytion analysis of simple substitution cyphers). It feels surprising that revers FFT can get a coheren…

As I understand it, the token embedding stream would be equivalent to multi-channel sampled waveforms. The model either needs to learn the embeddings by back-propagating through FFT and IFFT, or use some suitable tokenization scheme which the paper doesn't discuss (?).

It seems unlikely to work for language.

Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention

#124

Post by kridsdale1 »

evanb wrote: post #120
snickmy wrote: post #112

Earlier quoted context omitted.

this is a great way to put it, that said, it was not obvious to me that the attention space (how it is structured in LLMs) is a frequency domain

A cartoon: To form a coherent idea you need to coordinate a lot of tokens. In other words, ideas are long-distance correlations between tokens. Ideas are the long-wavelength features of streams of tokens. Is it exactly right? No. But as a cartoon it can motivate exploring an idea like this.

I like this. Anything that connects new synapses in my skull via analogy is a good post.

Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention

#125

Post by kridsdale1 »

ttoinou wrote: post #53
xeonmc wrote: post #7

Basically leverages convolution theorem[0]: expensive convolutions in direct space becomes simple multiplications in reciprocal space, and vice versa. Whereever you have a convolution operation on your data, transform them to the conjugate domain to turn it into multiplication. In other words, work in the domain that is natural to your data. [0] https://en.wikipedia.org/wiki/Convolution_theorem

Yeah basic math space transformation sandwich : 1) turn data into another space 2) operate in that space 3) transform back into original space. To optimize this, optimize each step and work as much as possible in the most efficient space

Remains to be seen how lossy the transformations are. We lose a lot of data in DSP (aka “media”) doing too much.

Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention

#126

Post by rocqua »

bigmattystyles wrote: post #78
xeonmc wrote: post #7

Basically leverages convolution theorem[0]: expensive convolutions in direct space becomes simple multiplications in reciprocal space, and vice versa. Whereever you have a convolution operation on your data, transform them to the conjugate domain to turn it into multiplication. In other words, work in the domain that is natural to your data. [0] https://en.wikipedia.org/wiki/Convolution_theorem

Is reciprocal space always just 1/space as in frequency=1/t?

Ite called reciprocal because the fourier transformation is it's own inverse, and the input and output space have the same 'shape' (functions from the reals to the complex numbers).

So they are considered two sides of the same coin. And reciprocal in that sense.

Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention

#127

Post by intalentive »

semiinfinitely wrote: post #122

I see no mention of prior work Hyena Operator which already demonstrated O(n log n) full context mixing several years ago. https://arxiv.org/abs/2302.10866

Hyena came out of Albert Gu's prior work in the same lab. https://arxiv.org/abs/2111.00396

Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention

#128

Post by hinkley »

I think in the age of telemetry we are also missing a substantial trick by not applying FFTs to cloud telemetry to smoke out epicycles and metastable systems before rather than after they trigger drama. This is unfortunately within my level of notice but not within my level of skill, and my dance card is already full.

"SLA's are most likely to be violated 23-25 minutes after a service deployment. Hmm, I wonder why that is... Oh no."

Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention

#129

Post by nimish »

ambicapter wrote: post #69
xeonmc wrote: post #7

Basically leverages convolution theorem[0]: expensive convolutions in direct space becomes simple multiplications in reciprocal space, and vice versa. Whereever you have a convolution operation on your data, transform them to the conjugate domain to turn it into multiplication. In other words, work in the domain that is natural to your data. [0] https://en.wikipedia.org/wiki/Convolution_theorem

> In other words, work in the domain that is natural to your data. Why would multiplication be more "natural" to a domain than convolution, as opposed to just simpler to calculate?

> just simpler to calculate

That's all that "natural" means in this context. It's like "elegant" -- if it just takes less effort to get the result u need then why wouldn't you take the easier route?

Post reply on HN