A hidden gem in sound symmetry

https://imgur.com/xLcvLIm

As you can see, the raw phase waveform is very "wavy", as might be expected. It oscillates rapidly, making it hard to see the patterns. But if you go to the tweets I linked above, you'll see the phase is much smoother in those images. How did I do it?

The key is to focus on every other line. Notice that if you simply pay attention to every odd row, it will be smooth.

I think I simply did "row 0, row 2, row 4, ... row n" followed by "row 1, row 3, row 5, ... row n + 1"

As for the fft of the fft trick for phase, I'm rsync'ing all of my old code and demo images to here:

https://battle.shawwn.com/sdb/voicecloning/

You may be interested in the png images, in particular the ones with "phase" in the names. You can probably ignore all the code except repl2.py.

Those images were generated via unknown methods -- sadly my repl sessions weren't saved. But, I happened to write down in repl2.py how the tweet images were generated:

  cv2.imwrite(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase-fft-abs.png"), np.abs(np.fft.fft2(-1+2*1/255*cv2.imread(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase.png")))))

  cv2.imwrite(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase-fft-abs2.png"), -1+2.0*np.abs(np.fft.fft2(-1+2*1/255*cv2.imread(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase.png")))))

Then, using the code above, the result: https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro...

I've verified that it still works. I think you can wget those images and copy-paste that code into a python repl.

So the only remaining question is, how was mel-phase-spectrogram-phase.png generated? Unfortunately that seems to be lost with the sands of time. But, as a hint, I think it was simply a matter of turning the phase component into x,y using atan2, then turning it into blue and red.

Also, completely unrelated, but I once made a super high resolution mel spectrogram that looked way cool and I can't resist showing it off: https://battle.shawwn.com/sdb/voicecloning/ultra-mel.png

I did all this when making 'Dr Kleiner sings "I Am the Very Model of a Modern Major General"' around a year ago.

https://www.youtube.com/watch?v=koU3L7WBz_s&ab_channel=Shawn...

Kinda funny that all of this visualization work was just to make memes, but the quest to meme turns out to be surprisingly motivating.

https://battle.shawwn.com/sdb/voicecloning/demo_output_101.w...

https://battle.shawwn.com/sdb/voicecloning/demo_output_75.wa...

Anyway, I think there's a lot left to discover in terms of audio visualization! I would definitely encourage you to play around with the phase component. The results can be pretty striking, as you can see from the "Result" image above (https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro...).

Sorry for the scattered explanation -- it's 4am here, but I wanted to give you some kind of writeup, even if it's rather disjointed. If you have more questions, be sure to ask! I can give better details tomorrow.

(Edit) After reading a bit more, it seems to make more sense that it's a combination of both, as the resonance on the cochlea is likely not 100% accurate, and conversely the impulses from peaks would tend to be around the areas of resonance, so rather than being mutually exclusive it makes sense these two effects work in parallel.

  Modern research suggests that the perception of pitch depends on both the places and patterns of neuron firings. Place theory may be dominant for higher frequencies.[4] However, it is also suggested that place theory may be dominant for low, resolved frequency harmonics, and that temporal theory may be dominant for high, unresolved frequency harmonics.[5]