The unreasonable effectiveness of the Fourier transform • Hacker News Board

Re: The unreasonable effectiveness of the Fourier transform

#151

Post by srean » Fri, Jan 09, 2026, 7:57 PM UTC

nihzm wrote: post #150

srean wrote: post #134

Earlier quoted context omitted.

You may well be right about neural networks. Sometimes models that seem nonlinear turns linear if those nonlinearities are pushed into the basis functions, so one can still hope. For GPT like models, I see sentences as trajectories in the embedded space. These trajectories look quite complicated and no obvious from their geometrical stand point. My hope is that if we get the coordinate system right, we may see someth…

> Sometimes models that seem nonlinear turns linear if those nonlinearities are pushed into the basis functions, so one can still hope. That idea was pushed to its limit by the Koopman operator theory. The argument sounds quite good at first, but unfortunately it can’t really work for all cases in its current formulation [1]. [1]: https://arxiv.org/abs/2407.08177

Quite so. Quite so indeed.

We know that under benign conditions and infinite dimensional basis must exist but finding it from finite samples is very non-trivial, we don't know how to do it in the general case.

Re: The unreasonable effectiveness of the Fourier transform

#152

Post by physicsguy » Fri, Jan 09, 2026, 8:29 PM UTC

hinkley wrote: post #147

My biggest missing feature for Grafana is that I want a Fourier transform that can identify epicycles in spikes of traffic. Like the first Monday of the month, or noon on tuesdays. I had a couple charts that showed a trend line of the last n days until someone in OPs noticed that three charts were fully half of our daily burn rate for Grafana. Oops. So I started showing a -7 days line instead, which helped me but con…

That wouldn't really work well because the sparsity of the periodic spikes wouldn't fit the assumption that the signal has a frequency component 'everywhere', even though it's periodic. You can see that mathematically - if you take the Fourier transform of an impulse signal you get a smeared result in frequency space.

You'd probably want to use a tool like calculating the cepstrum rather than fourier transform. Cepstral methods are commonly used in mechanical analysis to detect periodic impacts like where a gear tooth gets damaged.

Re: The unreasonable effectiveness of the Fourier transform

#153

Post by libraryofbabel » Fri, Jan 09, 2026, 9:21 PM UTC

hinkley wrote: post #148

libraryofbabel wrote: post #45

Earlier quoted context omitted.

One of the things I admire about many top mathematicians today like Terence Tao is that they are clearly excellent mentors to a long list of smart graduate students and are able to advance mathematics through their students as well as on their own. You can imagine a half-formed thought Terence Tao has while driving to work becoming a whole dissertation or series of papers if he throws it to the right person to work o…

Someone blew my mind by convincing me to read Bush’s “As we may think” which was published in 1945. Then I started digging into him and discovered he was also the second president of the ACM, was instrumental in shaping the formation of the National Science foundation (mainly by critiquing their initial plans as unworkable) and also Claude Shannon’s doctoral advisor. Because of course he was.

Not to mention instrumental in getting the Manhattan Project going, along with many other research projects during WWII. He basically knew everyone. I didn't know he was Shannon’s advisor though!

Re: The unreasonable effectiveness of the Fourier transform

#154

Post by ggm » Fri, Jan 09, 2026, 9:46 PM UTC

refulgentis wrote: post #139

ambicapter wrote: post #129

Earlier quoted context omitted.

What does "She isn't wings" mean?

Paul McCartney started a band called Wings and she was also in it. I think the idea is "she received nepotistic spoils, lived in the shadow of dad even in his backup projects that 0.01% of people who know the Beatles even recognize." (This elides a very successful career as a fashion designer, as well as the awkward question of what _would_ have guaranteed her more “success”, as well as a lack of understanding of how…

I meant, she isn't defined by wings and being daddy's kid. She struck out into a field neither Paul nor Linda had occupied. She isn't (defined by) wings.

Re: The unreasonable effectiveness of the Fourier transform

#155

Post by madhadron » Fri, Jan 09, 2026, 10:39 PM UTC

srean wrote: post #134

madhadron wrote: post #120

Earlier quoted context omitted.

All these transforms are switching to an eigenbasis of some differential operator (that usually corresponds to a differential equation of interest). Spherical harmonics, Bessel and Henkel functions, which are the radial versions of sines/cosines and complex exponential, respectively, and on and on. The next big jumps were to collections of functions not parameterized by subsets of R^n. Wavelets use a tree shapes para…

You may well be right about neural networks. Sometimes models that seem nonlinear turns linear if those nonlinearities are pushed into the basis functions, so one can still hope. For GPT like models, I see sentences as trajectories in the embedded space. These trajectories look quite complicated and no obvious from their geometrical stand point. My hope is that if we get the coordinate system right, we may see someth…

I’m not sure what you mean by a change of basis making a nonlinear system linear. A linear system is one where solutions add as elements of a vector space. That’s true no matter what basis you express it in.

Re: The unreasonable effectiveness of the Fourier transform

#156

Post by bcrl » Sat, Jan 10, 2026, 12:13 AM UTC

hinkley wrote: post #149

3eb7988a1663 wrote: post #94

Earlier quoted context omitted.

MIT was able to reconstruct voice by filming a bag of chips on a 60FPS camera. I would hesitate to say how much information can leak through. https://news.mit.edu/2014/algorithm-recovers-speech-from-vib...

I befriended the guy in high school who built a Tesla coil. For his next trick he was building a laser to read sound off of plate glass. The decoder was basically an AM radio. Which high school me found slightly disappointing.

I basically asked my math and physics teachers in high school what the Fourier transform was, but none of them knew how to answer my questions (which were about digital signal processing -- modems were important things to us back in the early '90s). If I had to do it over again, I would have audited the local university's electrical engineering and math courses in evenings. The first time MIT ran 6002x online back in 2012, the course finally answered a lot of those questions when touching upon filters and bandwidth.

Re: The unreasonable effectiveness of the Fourier transform

#157

Post by RossBencina » Sat, Jan 10, 2026, 12:56 AM UTC

srean wrote: post #113

People go all dopey eyed about "frequency space", that's a red herring. The take away should be that a problem centric coordinate system is enormously helpful. After all, what Copernicus showed is that the mind bogglingly complicated motion of planets become a whole lot simpler if you change the coordinate system. Ptolemaic model of epicycles were an adhoc form of Fourier analysis - decomposing periodic motions over…

> exponential functions remain (scaled) exponential when passed through such operations.

See also: eigenvalue, differential operator, diagonalisation, modal analysis

Re: The unreasonable effectiveness of the Fourier transform

#158

Post by anamax » Sat, Jan 10, 2026, 1:02 AM UTC

srean wrote: post #113

People go all dopey eyed about "frequency space", that's a red herring. The take away should be that a problem centric coordinate system is enormously helpful. After all, what Copernicus showed is that the mind bogglingly complicated motion of planets become a whole lot simpler if you change the coordinate system. Ptolemaic model of epicycles were an adhoc form of Fourier analysis - decomposing periodic motions over…

> My suspicion is that we are in Ptolemaic state as far as GPT like models are concerned. We will eventually understand them better once we figure out what's the better coordinate system to think about their dynamics in.

Most deep learning systems are learned matrices that are multiplied by "problem-instance" data matrices to produce a prediction matrix. The time to do said matrix-multiplication is data-independent (assuming that the time to do multiply-adds is data-independent).

If you multiply both sides by the inverse of the learned matrix, you get an equation where finding the prediction matrix is a solving problem, where the time to solve is data dependent.

Interestingly enough, that time is sort-of proportional to the difficulty of the problem for said data.

Perhaps more interesting is that the inverse matrix seems to have row artifacts that look like things in the training data.

These observations are due to Tsvi Achler.

Re: The unreasonable effectiveness of the Fourier transform

#159

Post by hedgehog » Sat, Jan 10, 2026, 2:05 AM UTC

bcrl wrote: post #156

hinkley wrote: post #149

Earlier quoted context omitted.

I befriended the guy in high school who built a Tesla coil. For his next trick he was building a laser to read sound off of plate glass. The decoder was basically an AM radio. Which high school me found slightly disappointing.

I basically asked my math and physics teachers in high school what the Fourier transform was, but none of them knew how to answer my questions (which were about digital signal processing -- modems were important things to us back in the early '90s). If I had to do it over again, I would have audited the local university's electrical engineering and math courses in evenings. The first time MIT ran 6002x online back in…

Yeah I wish I had known about or had access to that stuff when I was a kid. To really learn and internalize ideas like negative frequency early would have been quite fun.

Re: The unreasonable effectiveness of the Fourier transform

#160

Post by ggm » Sat, Jan 10, 2026, 4:03 AM UTC

ambicapter wrote: post #128

ggm wrote: post #75

Earlier quoted context omitted.

But what he deemed being posited as true, was this really bad parenting? It could be to head off competition or it could be brutal realism to head off future depression. Nepotism existed since time immemorial but for a mathematical genius, what was the nepotistic deliverable for the child? A sinecure placement at university?

> But what he deemed being posited as true Implicit in the "correctness" of this motive is the idea that unless you're #1 in your field, you are nothing (depression implies strong feelings of worthlessness). I don't know if you think that's a great lesson to teach your kids as a parent, but I don't.

A valid point. I might add I followed my dad into the field and my son followed me into the field and both academically and in capability he exceeds my ability. I'm enormously proud of him, and I did not seek to prevent him following his heart into the field beyond noting there are other dreams to follow worth considering. I did seek to persuade him not to become a chef, it's a really hard, stressful life.

I totally did not exceed my dads abilities: as one of the 1950s computing pioneers who build some of the first machines in the UK, founded a department of computer science and did work in IFIP, and the BCS, this was always going to be hard.