Synchronizing Pong to music with constrained optimization

> Synchronizing pong to music with constrained optimization

Nothing new. Apparently there are references to people doing this in ancient and medieval times.

https://en.wikipedia.org/wiki/Flatulist

This is really cool. I took an optimization class a few years back, but haven't made the time to do anything fun with it since. This inspires me to do it.

I do kind of wish that the last note corresponded to a game over, though, and I wonder if a smaller screen or faster ball would widen the playing field a little. Maybe I'll fork the code and try some of those out myself.

I'm a novice at machine learning, but Open AI made a python library for reinforcement learning in video games, and a fork of it is still actively maintained [1]. It's been a few years, but I remember being able to get it up and running in a day or two (maybe a weekend). It used the Retroarch emulator, which is compatible with a huge number of emulators and consoles.

https://github.com/Farama-Foundation/Gymnasium

There also SethBling's excellent video on YouTube about machine learning specifically with Super Mario World:

https://www.youtube.com/watch?v=qv6UVOQ0F44

I encourage you to give it a try! I feel that video games are a bit underrated by current AI buzz, I think there's lots of potential for a machine to learn a skill through playing a game. And lots of potential for a game being selected or even created with the goal of teaching a specific skill. However at that point maybe it's better to forego the audio and visuals and speak to the machine in text or with pure data.

On the other hand, I have seen a video about convolutional neural networks that feed on each pixel of an image. So perhaps training with sound data, or with the pixels of a spectrogram, could have some positive results. And certainly I would be amused to see a game played in time with music, or possibly even dancing with a songs melody, harmony, and story as well.

Anything that's ever been created by humans, existed first in the imagination of a human brain. And you've got one of those. A mental vision pursued and brought from the mind into physical reality is a beautiful thing, a gift for all of humanity in my eyes. I think it's quite worthwhile. But that's just my perspective. Thank you for sharing your imagination. Have a nice day

If only operational research courses could have been so fun many years ago… excellent write-up!

Having N paddles would be cool as well, perhaps with a cost for paddles getting too close (to avoid trivialities), and optimizing which note each paddle represents/beats, maybe imposing a cost on drastic pitch changes so that each specializes in a pitch range.