My EE education keeps coming back.
The FFT Strikes Back: An Efficient Alternative to Self-Attention
161–170 of 175 posts
Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention
#162Earlier quoted context omitted.
Is reciprocal space always just 1/space as in frequency=1/t?
In the case of FFTs, no. Which is why I prefer the term Fourier space. I don’t like frequency domain because I frequently work with 3-D and 5–D FFTs while I’ve always felt frequency is connected to single dimension FFT.
Maybe something like "recurrence space" would be better. Frequency does have a physical interpretation which could be misconstrued, e.g. the FFT of a wave in the space domain yields the wavenumber in the independent variable, not the frequency.
Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention
#163Earlier quoted context omitted.
Or for that matter, a transform that's learned from the data :) A neural net for the transform itself!
That would be super cool if it works! I’ve also wondered the same thing about activation functions. Why not let the algorithm learn the activation function?
This idea exists (the broad field is called neural architecture search), although you have to parameterize it somehow to allow gradient descent to happen.
Here are examples:
Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention
#164I sort of grasp big O notation...but this is sort of over my head like most stuff that has to do with computer or electrical engineering. As someone who is absolutely terrible at math, I envy the people who grasp or at least can learn this type of stuff and get an engineering degree and license. All I really know about FFT is that is changes a signal, its somehow used in processing signals of some kind, and it appare…
The simple explanation: Lets say you have a one-dimensional time-domain signal, e.g. an audio signal measured by a microphone, which measures the displacement of air vs. time at a fixed point (assuming you dont move the microphone I guess)
The Fourier transform (of which the FFT is a discrete version of) decomposes that 1D time-domain signal (e.g. an audio signal, time vs. displacement) into frequency vs. magnitude and phase components
The frequency is basically the pitch. So for a pure sine wave, or pure tone - which sounds like those "off air" TV signals we used to get late at night back in the day. You get a bunch of zeros and a single "spike" at the frequency of the tone. The larger the amplitude of the signal, the larger the magnitude of the spike will be. As the pitch (frequency) increases/decreases, the location of this spike moves up/down along horizontal axis
The phase is basically the time offset of the signal. A tone which was delayed somehow will show up as a different phase. Note this is a relative measure - not absolute. So you won't be able to tell if the signal was offset by 1s or 2s, etc. because it has units of radians(angle), which have to "reset" as the angle wraps around the circle.
So for one signal (time vs. amplitude), you actually get two pieces of information (frequency vs. magnitude/phase)
However if you understand imaginary numbers/complex variables, those two signals are really just the magnitude and argument of FFT output, which produces a complex function
Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention
#165Makes me think what if NNs are treated as black box signal processing units. What other techniques can we borrow from signal processing?
In image processing at least, NN typically learn a Fourier or Wavelet representation in their first layers. Biggest benefit of applying a transformation beforehands is to reduce training time / obtain better generalization by "removing the dimension that doesn't matter". E.g. in a suitable space, one coordinate could represent the rotation of an object. You could do the transform and discard this dimension if your NN…
In image processing I thought there was a whole host of specialized algorithms, such as edge detection, SCC, etc. that were run before the data was even fed into the ANN.
Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention
#166Earlier quoted context omitted.
Overall, i like this guys papers but they strike me as someone who is very smart but hasnt looked through the literature carefully. Many of the techniques he is proposing were already done about 5-6 years ago. However, I imagine that because the field is flooded with new humans, they are not aware of this research or think it will lead to a fruitful end (which many other researchers have already thought of this and i…
Reviving old ideas and comparing them to SOTA is not necessarily bad especially if they provide benefits over the SOTA model. It brings the old ideas into the community idea cache if you will. It’s somewhat annoying if the authors do it thinking it’s a novel idea when it fact it’s a 20-30 year old one. This reminds me of some HN comments about rocketry ideas and in the thread one of the comments was “Everything in ro…
Yes, but they should at least reference the older papers and explain "whats new"
Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention
#167Earlier quoted context omitted.
Or for that matter, a transform that's learned from the data :) A neural net for the transform itself!
That would be super cool if it works! I’ve also wondered the same thing about activation functions. Why not let the algorithm learn the activation function?
Mostly because of computational efficiency irrc, the non linearity doesn’t seem to have much impact, so picking one that’s fast is a more efficient use of limited computational resources.
Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention
#168[deleted]
Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention
#169The Fourier transform is taken along the “token” dimension. However, in many applications, this dimension is not meaningful. That’s why transformers are a great option for consuming data which is permutation invariant. I would like to see additional experiments using the lesser known Fourier transform over finite groups [1], which is permutation invariant but shares many properties with the standard Fourier transform…
Not an expert in this space. Aren't tokens transformed with position dependent information in most models? I believe llama applies a rotation to the vector based on the position in the input.
That's true in the realm of LLMs. But even in this case, the position information is added only into the first layer. Tokens in later layers can choose to "forget" this information. In addition there are applications of transformers in other domains. See https://github.com/cvg/LightGlue or https://facebookresearch.github.io/3detr/
Re: The FFT Strikes Back: An Efficient Alternative to Self-Attention
#170I think in the age of telemetry we are also missing a substantial trick by not applying FFTs to cloud telemetry to smoke out epicycles and metastable systems before rather than after they trigger drama. This is unfortunately within my level of notice but not within my level of skill, and my dance card is already full. "SLA's are most likely to be violated 23-25 minutes after a service deployment. Hmm, I wonder why th…
"I'm afraid I cannot deploy your application, Dave"
Jokes aside one area this could be really worth money is predicting cycles of traffic and saving with ramp up and ramp down of server instances. It's the kind of work that if you're doing it out of your time the company would never give you greenlight it but if you pack it as a shelf product they would totally buy it.