Live data from Hacker News

Synchronizing Pong to music with constrained optimization

victortao.substack.com

31–40 of 40 posts

Re: Synchronizing Pong to music with constrained optimization

#31

Very cool! As a further variation on this idea, I'm imagining training a reinforcement learning agent on atari games / super mario, but with an additional music-based reward/input, to try to get a "musical" looking playthrough... (Not sure how good it would look / whether it would be worth it though...)

[deleted]

Re: Synchronizing Pong to music with constrained optimization

#32
Atari had a video music "visualizer" device back in the late 1970s. Designed by one of the developers of the Pong game. One of if not the first consumer product of its kind.

https://en.wikipedia.org/wiki/Atari_Video_Music

If you've seen the movie Over the Edge, Claude and Johnny have one at their house.

Re: Synchronizing Pong to music with constrained optimization

#33
While technically okay, there are multiple cases where a paddle and the ball move at the almost identical speed and it looks like the paddle is pushing the ball all the time. (By the way, p[i] = 0 should be disallowed for this reason.) This is inevitable when a large d[i] is immediately followed by a very small d[i+1], but it might be able to avoid them whenever feasible.

Re: Synchronizing Pong to music with constrained optimization

#34

While technically okay, there are multiple cases where a paddle and the ball move at the almost identical speed and it looks like the paddle is pushing the ball all the time. (By the way, p[i] = 0 should be disallowed for this reason.) This is inevitable when a large d[i] is immediately followed by a very small d[i+1], but it might be able to avoid them whenever feasible.

You could penalize the objective function proportionally to d'[i]

Re: Synchronizing Pong to music with constrained optimization

#36

Very cool! As a further variation on this idea, I'm imagining training a reinforcement learning agent on atari games / super mario, but with an additional music-based reward/input, to try to get a "musical" looking playthrough... (Not sure how good it would look / whether it would be worth it though...)

Not very good. Mario doesn’t have enough rhythm to evoke a lot of musicality.

The original game’s sound was tied to the frame rate so this vaguely happened by default. Later ports to PAL broke this because it ran at a slower frame rate.

Re: Synchronizing Pong to music with constrained optimization

#38
post #26

This reminds a bit of a project I coded the audio for an art exhibition around 20 years ago, which was for multiple simultaneous players. As the game got more intense it became apparent that the ball/wall sounds were playing music, and that the three players were all actually playing one musical piece. It was based around 3 arcade cabinets pointing together, so the players couldn't see what was on each others screens…

That’s neat. So it was a static composition that the gameplay was tweaked into fitting?

Re: Synchronizing Pong to music with constrained optimization

#40
post #38
post #26

This reminds a bit of a project I coded the audio for an art exhibition around 20 years ago, which was for multiple simultaneous players. As the game got more intense it became apparent that the ball/wall sounds were playing music, and that the three players were all actually playing one musical piece. It was based around 3 arcade cabinets pointing together, so the players couldn't see what was on each others screens…

That’s neat. So it was a static composition that the gameplay was tweaked into fitting?

Somewhere in between. It was a static composition (as in, it had a start and an end) but there were multiple loops per instrument, so more competent players would reach a more frenetic musical performance of the piece if they kept more of the balls in play.

The audio engine (the bit I worked on) was in effect a stem based mixer, but with a state transition diagram per stem, with multiple loops available. Depending on the route through the transitions (triggered by events from the game play, for example, how many balls the player is keeping in play) it was possible to reach more complex or simpler performances of the same musical piece, so the players decisions and ability would affect how the performance of the music, not the music itself if that makes sense?

Post reply on HN