Synchronizing Pong to music with constrained optimization
victortao.substack.com
Synchronizing Pong to music with constrained optimization
1–10 of 40 posts
Re: Synchronizing Pong to music with constrained optimization
#2pretty neat! it feels like if you spaced out “important” beats instead of most of them and shrunk the play area so the paddles are larger, it would have an even more interesting effect.
Re: Synchronizing Pong to music with constrained optimization
#3Absolutely wonderful!
> "We obtain these times from MIDI files, though in the future I’d like to explore more automated ways of extracting them from audio."
Same here. In case it helps: I suspect a suitable option is (python libs) Spleeter (https://github.com/deezer/spleeter) to split stems and Librosa (https://github.com/librosa/librosa) for beat times. I haven't ventured into this yet though so I may be off. My ultimate goal is to be able to do it 'on the fly', i.e. in a live music setting being able to generate visualisations a couple of seconds ahead being played along with the track.
Not sure if this is unsavory self promotion (it's not for commercial purposes, just experimenting), but I am in the middle of documenting something similar at the moment.
Experiments #1 - A Mutating Maurer Rose | Syncing Scripted Geometric Patterns to Music: https://www.youtube.com/watch?v=bfU58rBInpw
It generates a mutating Maurer Rose using react-native-svg on my RN stack, synced to a music track I created in Suno AI *. Manually scripted to sync up at the moment (not automatic until I investigate the above python libs).
Not yet optimised, proof of concept. The Geometric pattern (left) is the only component intended to be 'user facing' in the live version - But the manual controls (middle) and the svg+path html tags (right) are included in this demo in order to show some of the 'behind the scenes'.
Code not yet available, app not yet available to play with. Other geometric patterns in the app that I have implemented:
- Modified Maurer
- Cosine Rose Curve
- Modified Rose Curve
- Cochleoid Spiral
- Lissajous Curve
- Hypotrochoid Spirograph
- Epitrochoid Spirograph
- Lorenz Attractor
- Dragon Curve
- Two Pendulum Harmonograph
- Three Pendulum Harmonograph
- Four Pendulum Harmonograph
This is the Typescript Maurer Rose function (that is used with setInterval + an object array of beat times which determine when to advance the 'n' variable):
export const generateGeometricsSimplemaurer = (n: number, d: number, scale: number = 1) => {
const pathArray: TypeSvgPathArray = [];
for (let i = 0; i The geometric patterns (with their controls) will have a playground app that you can use to adjust variables... As for the music sync side, it will probably take me a long time.
*Edit: I just noticed that the author (Victor Tao) actually works at Suno
Re: Synchronizing Pong to music with constrained optimization
#4Really nice stuff. I cannot send a heart without subscribing which doesnt feel right for me.
Re: Synchronizing Pong to music with constrained optimization
#5Awesome work bro! Your company is hiring? I'd be super thrilled to work with you.
Re: Synchronizing Pong to music with constrained optimization
#6This is so freaking cool! I was mesmerized watching the paddles move as the beat progressed. There are certain things that just look right which makes it beautiful.This project is one of them!
Re: Synchronizing Pong to music with constrained optimization
#7Prior art: Eisenfunk - Pong (https://www.youtube.com/watch?v=cNAdtkSjSps)
Re: Synchronizing Pong to music with constrained optimization
#8Prior art: Eisenfunk - Pong ( https://www.youtube.com/watch?v=cNAdtkSjSps )
Bit different though! In your example, the video is made from manually syncing with the song bpm, as the beep is at a constant rate. It's basically just a hand-made visualization of (every other) kick drum.
While the submission has the notes not at a basic 1/4 tempo, and is automatically "animated" based on the constrained optimization. Also leads to a much more interesting visualization :)
Re: Synchronizing Pong to music with constrained optimization
#9Prior art: Eisenfunk - Pong ( https://www.youtube.com/watch?v=cNAdtkSjSps )
Bit different though! In your example, the video is made from manually syncing with the song bpm, as the beep is at a constant rate. It's basically just a hand-made visualization of (every other) kick drum. While the submission has the notes not at a basic 1/4 tempo, and is automatically "animated" based on the constrained optimization. Also leads to a much more interesting visualization :)
No constraint optimization can replace Pentafunk Jenny ;)
Re: Synchronizing Pong to music with constrained optimization
#10Awesome work!
How is the beat used to sync the pong chosen? Like for Bad Apple!, especially around 1m55 https://www.youtube.com/watch?v=bvxc6m-Yr0E it seems off
Good suggestion from a YouTube commenter, pasting it here
> This is pretty cool.. it would be cooler if there were multiple pongs and paddles for each type of beat (like high beats and low beats)