Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A hidden gem in sound symmetry

soundshader.github.io

Post reply on HN
Print view View original

31–40 of 54 posts

Re: A hidden gem in sound symmetry

#31

Post by gbh444g »

MauranKilom wrote: post #18

I'm very interested in this, but I can't seem to get the demo page working in latest Firefox: > AudioContext.createMediaStreamSource: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. Edge worked though. Didn't try Chrome. Edit: This is beautiful! I have checked out a variety of songs, and I feel the visualization is rather dominated by whatever frequency is loudest. (E.g…

That's indeed the number 1 problem with the visualizer. ACF waves can greatly vary in magnitude and if there is a loud bass wave that's 1000x bigger than small wavelets from background music, those wavelets will be present, but barely visible. That's also why classical or otherwise "peaceful" music looks so good: all waves have about the same height.

I've in fact tried implementing the equal loudness contour - try adding ?acf.aweight=1 to the URL. However the result is mediocre. I've also tried applying a few bandpass filters for low, mid and high frequency ranges, rendering them separately with different colors and then mixing images together. The result is, again, medicore. I've been entertaining the idea that ACF waves are ought to be rendered like ocean waves: via light reflections.

Re: A hidden gem in sound symmetry

#32

Post by gbh444g »

hiisukun wrote: post #29

Somehow I have memories of the 1990s and audio visualisations coming back... Fantastic. I'd love to be able to plug this into mpv for playing audio files. I wonder how hard this would be (or how many resources it would use...). I really enjoyed your work: the write-up was clear and the demo page worked well.

For visualization, you'd want to take at least 20 sound samples per second, each sample about 0.1 sec long, so they would overlap a bit to capture freq/amp modulations. With the typical 44,100 Hz sample rate, 0.1 sec would be an array of 4410 float32s. Round that down to 4096. Then you run FFT twice over this input: that's Nlog2(N), N=4096. So you have about KN log2(N) ops/sec, with K=20, N=4096 that's 1 million flops. Updating pixels of the 1000x1000 image would be another 1 Mflops. Well, brightness adjustment to the 3 sigma range is another pass and another 1 Mflops. I know for a fact that this thing can draw 1024x1024 images at 60 fps on pure javascript, but if you don't like the noise of CPU fan, you'll have to write it in GLSL.

Re: A hidden gem in sound symmetry

#33

Post by hashtagmarkup »

gbh444g wrote: post #20
hashtagmarkup wrote: post #14

Earlier quoted context omitted.

Are you familiar with the "Circle of fifths"? That's the first thing that came to mind when I saw your circle patterns, and it directly relates to "harmony".

ACF composes all harmonics together and that's what likely makes the images visually appealing. However ACF doesn't give special treatment to harmonics that are exactly N octaves apart, e.g. A4 and A7 notes.

Would ACF be better if did give special treatment to those harmonics? I understand they are an arbitrary distinction, but humans do seem to like them.

Re: A hidden gem in sound symmetry

#34

Post by gbh444g »

hashtagmarkup wrote: post #33
gbh444g wrote: post #20

Earlier quoted context omitted.

ACF composes all harmonics together and that's what likely makes the images visually appealing. However ACF doesn't give special treatment to harmonics that are exactly N octaves apart, e.g. A4 and A7 notes.

Would ACF be better if did give special treatment to those harmonics? I understand they are an arbitrary distinction, but humans do seem to like them.

It would. The 12 notes are usually mapped to 12 colors, and ideally the sound image would reflect that. One "brute force" way to do that is to split the spectrum into 12 parts, draw 12 ACF images and then mix them. A less brute force approach is to tweak ACF to recognize that F-2F-4F-etc frequencies are specially related, even more specially than just F-2F-3F-4F-etc.

Re: A hidden gem in sound symmetry

#35

Post by acjohnson55 »

klik99 wrote: post #11
gbh444g wrote: post #2

Hi HN, author here. A few comments on how I came up with this idea. I've been trying to find a "proper" connection between audible sound and visible shape, a connection that would not only preserve all the information, but would also properly visualize the "symmetry" in sound, so that messy sound would turn into messy images and harmonic sound would turn into visually appealing images. The latter part is hard, as per…

In your example saying that the ear doesn’t work as an FFT, I think you’re confusing physical processes (the ear) with perceptual processes (psychoacoustics and interaction between the brain and ear). The cochlea itself is a physical FT, based on resonances in the thickness at various points. That gets passed to the brain as basically sine-frequency data. The autocorrelation part comes in when the brain processes thi…

IIRC, that's probably not quite true. It's been about a decade since I studied hearing, but my understanding is that the idea of the cochlea being a physical FT (e.g. the place theory of hearing) doesn't explain some phenomena, which also probably aren't psychoacoustic.

https://en.m.wikipedia.org/wiki/Temporal_theory_(hearing)

Re: A hidden gem in sound symmetry

#36

Post by alanbernstein »

gbh444g wrote: post #2

Hi HN, author here. A few comments on how I came up with this idea. I've been trying to find a "proper" connection between audible sound and visible shape, a connection that would not only preserve all the information, but would also properly visualize the "symmetry" in sound, so that messy sound would turn into messy images and harmonic sound would turn into visually appealing images. The latter part is hard, as per…

I like the vaguely mathematical connotation of the word "ornament" here, I have never encountered that as a technical term before. It makes me realize that ornament is a good description for many of my favorite mathematical concepts/structures.

Your comment reminds me of the search for the mandelbulb fractal. It seems a fitting comparison, a bulb being a sort of ornament.

Anyway, interesting work.

Re: A hidden gem in sound symmetry

#37

Post by gbh444g »

alanbernstein wrote: post #36
gbh444g wrote: post #2

Hi HN, author here. A few comments on how I came up with this idea. I've been trying to find a "proper" connection between audible sound and visible shape, a connection that would not only preserve all the information, but would also properly visualize the "symmetry" in sound, so that messy sound would turn into messy images and harmonic sound would turn into visually appealing images. The latter part is hard, as per…

I like the vaguely mathematical connotation of the word "ornament" here, I have never encountered that as a technical term before. It makes me realize that ornament is a good description for many of my favorite mathematical concepts/structures. Your comment reminds me of the search for the mandelbulb fractal. It seems a fitting comparison, a bulb being a sort of ornament. Anyway, interesting work.

An ornament is something that's almost invariant under some transforms. For example, these ACF images are almost invariant under some rotations. Sound waves are almost invariant under temporal shifts. I'd argue that what makes an ornament look good is the ease of recognizing those transforms and what makes sound sound sound is the ease of recognizing those temporal shifts.

Re: A hidden gem in sound symmetry

#38

Post by sillysaurusx »

gbh444g wrote: post #2

Hi HN, author here. A few comments on how I came up with this idea. I've been trying to find a "proper" connection between audible sound and visible shape, a connection that would not only preserve all the information, but would also properly visualize the "symmetry" in sound, so that messy sound would turn into messy images and harmonic sound would turn into visually appealing images. The latter part is hard, as per…

Surprisingly, I stumbled across a result that seemed similar yet weirdly different: https://twitter.com/theshawwn/status/1176070857468329984?s=2...

I take the FFT of the phase component, which is very similar to ACF; it’s the FFT of an FFT, but preserves phase. It even takes abs(), which might be mostly equivalent to your squaring operation.

Weird. I am really not trying to claim that I discovered ACF — quite the opposite. My result was shockingly different to what you found, even though the operations are so close to identical.

I think phase is extremely important in visualization. You can see why here: https://twitter.com/theshawwn/status/1176070853819342848?s=2...

You’ve come up with one of the most gorgeous visualizations I’ve ever seen for signals in general!

One way to incorporate phase: turn the angle into an x,y coordinate using atan2, then shade red and blue based on x and y. E.g. x of 1.0 is “full red”, x of -1.0 is no red; ditto for y, but with blue.

The other trick I used was to un-interleave the lines. Basically I noticed that every other line has a strong correlation; therefore drop every even numbered line to remove the aliasing artifacts. Then suddenly you get nice and smooth phase interpolations.

Re: A hidden gem in sound symmetry

#39

Post by gbh444g »

sillysaurusx wrote: post #38
gbh444g wrote: post #2

Hi HN, author here. A few comments on how I came up with this idea. I've been trying to find a "proper" connection between audible sound and visible shape, a connection that would not only preserve all the information, but would also properly visualize the "symmetry" in sound, so that messy sound would turn into messy images and harmonic sound would turn into visually appealing images. The latter part is hard, as per…

Surprisingly, I stumbled across a result that seemed similar yet weirdly different: https://twitter.com/theshawwn/status/1176070857468329984?s=2... I take the FFT of the phase component, which is very similar to ACF; it’s the FFT of an FFT, but preserves phase. It even takes abs(), which might be mostly equivalent to your squaring operation. Weird. I am really not trying to claim that I discovered ACF — quite the opp…

Interesting. Figuring out the phase problem is one of my biggest TODO items.

How did you compute FFT of the phase? The thing is, phase is discontinuous or multivalued function if we represent phase as a real number. We could also represent phase as a complex number of unit magnitude: exp(i phi). It would be continuous, but complex-valued.

And phase is indeed important for hearing:

https://auditoryneuroscience.com/vocalizations-speech/speech...

I didn't quite get the trick with uninterleaving the lines.

Post reply on HN