ProjectM – The most advanced open-source music visualizer

Indeed! I produce music for fun using virtual synths, multi-layer sound libraries, audio plug-ins and MIDI combined in a software digital audio workstation (Cubase at the moment). I find it super enjoyable and despite not formally studying music, not reading traditional music notation nor being able to play any musical instrument well in real-time, through step recording to a MIDI multitrack timeline (and subsequent on-screen tweaking) I manage to create remarkably pleasing, quite commercial-sounding original music.

As an equally enthusiastic video hobbyist, I've long-wanted to create my own original visuals to accompany my music. Yet the "visual synthesis" tools I've played with seem to remain split into two extremes - essentially "too algorithmic / Geiss-like" on one end and "too complex and art-talent-thirsty" on the other (for example Blender 3D, AfterEffects). A similar gap existed in music tools 20 years ago but remarkable progress has been made in closing that gap empowering in-betweeners like me to achieve wonderful results with only hobby-level skills, time and budgets. Sadly, that stunning progress in democratizing music synthesis has not been equally mirrored in the visual synthesis domain. The visual side mostly remains polarized into DJ/VJ-centric real-time apps and pro-level video editing and 2D/3D rendering tools.

It's odd because the musical synthesis domain seems to have been unleashed by increasing access to dedicated hardware DSPs in the 90s followed by rapid, iterative 'software-ization' as general-purpose processors became capable of replacing DSPs (MMX etc). GPUs have followed a similar march down the curve of democratization but for some reason broad access hasn't unleashed a similar hobby renaissance on the visual synthesis side like it has for visual capture and editing with mobile/DSLR imaging and desktop video editing.

However, I still remain hopeful that perhaps the visual end of the digital synthesis revolution has only been delayed. I find encouraging signs in the real-time rendering and compositing capabilities of GPUs driven by game engines from the nascent efforts of Machinema a decade ago to, more recently, ILM using Unreal Engine to create remarkable visuals for the Disney/Star Wars series "The Mandalorian" using conceptually approachable live projection mapping techniques. The custom code ILM paid Unreal to add has since been bundled (unsupported) into all versions of the engine including the personal/hobby-use ones and garage experiments can already be found on YouTube.

As for providing more useful audio sources as input to the visual pipeline, there are millions of desktop music hobbyists ranging from Garagebanders on tablets and phones to DAW users on desktop. All of them are already capable of supplying separated, uncompressed multi-track audio stems (pre or post downstream effects like EQ/reverb) along with expressive metadata in the form of MIDI 2 notes and articulations. There are even a variety of standards across Windows, Mac, iOS and Android for real-time, multi-track transfer between applications within a device. For distribution formats, new consumer digital digital audio interfaces from HDMI 2.x on the interconnect side to lossless multi- and N-channel spatial audio standards on the encoding side are democratizing the necessary I/O capabilities.

I watched the demo video, and couldn't convince myself that the images had anything to do with the accompanying music. Cool animations, but not a "music visualizer."

I... surely you're joking. The Steam Deck poses no serious threat to Nintendo.

It's not remotely possible for Steam Deck to have the sort of first-party integration Nintendo accomplishes around co-op games, multiple controllers in multiple configurations, motion controls, pairing, latency, suspend/resume...

Like, have you used a Valve product before? They're beautiful and inspiring but... when they work, it's still not Nintendo quality.

The vast, vast majority of Nintendo customers know nothing of game mods or competitive multiplayer or care about latency or netcode. I doubt the majority even play Smash or any fighting game.

VJ's tend to use multiple different apps and connect them using syphon (Mac) or spout (windows) to share textures. I've seen resolume (clip heavy content) and touchdesigner (node based "visual IDE" ) mentioned. Those are all big in the space and offer something unique.

I wanted to add https://www.synesthesia.live to that list. It has a ton of audio reactive content and the best audio reactivity in the space.

But the key really is in using other apps + syphon/spout to combine all the textures into unique content. Then manipulating that in something like touch designer.

Full disclosure: I'm one of the creators of synesthesia.live but use multiple apps, Touch Designer everyday.