Author here. Surprised to see this here as I've been literally spending the last 2 weeks on a complete rewrite from scratch that actually layers REAL text over the blocks. You can get an idea of my approach with this function: https://github.com/tombh/texttop/blob/webext-rewrite/webext/... After the success of hitting the front page here last year, I really wanted to sit down and do this properly. So not only am I wo…
Either something like Opus, or...
You could do a decomposition of the audio signal and send a small array of {frequency, amplitude} pairs for each frame, then ramp to each new pair as they arrive.
If you do this fast enough then you get just enough of the timbre of the original to make a listener think they are hearing the "right" thing for a video they already know.
Even with a single oscillator you can get some interesting sounds this way.
edit: syntax error