Digital Audio Workstation Front End Development Struggles

Its weird to see the impressions of someone who has apparently never used the equipment used in music production trying to talk intelligently about said devices. So many incorrect assumptions its really painful to read as someone aquainted with audio engineering. Interesting window into how the devs see it nonetheless.

I actually know this space fairly well, probably better than most, because being an implementor of a UI compositor raises some difficult technical challenges with respect to UI tree backing layers and draw invalidation.

I'm still not actually sure what the state of the art on these matters is after having researched this for years, and I suspect even what we do today across software like web browsers, video game UI is wildly inefficient for modern rich user-interface requirements.

For instance, you could tailor GUI software to strictly draw to specific frame buffer/backing layer regions and you could manually invalidate and redraw when the developer specified, but that's generally far too implementation specific for most UI developers to actually follow. A master UI implementor wouldn't have too much trouble with this, but most people don't have a decade of graphics API level experience on top of designing world-class user-interfaces.

So you have some ambiguous UI tree you need to draw and redraw when things move or animate. Today's state of the art is to multithread draw the space to tiles using virtual graphics commands that later get converted into target graphic API calls and those tiles translate to the corresponding graphics API backing layer abstraction, some sort of frame buffer.

When the UI tree changes state, those leaves are invalidated all the way up/down the tree to the root at which point the compositor redraws what's necessary and combines the layers into what gets displayed on the screen.

There's a bit of work to separate disparate UI element bounds from layers which require acceleration, but those nodes are flagged for the requirement usually by explicit properties.

This is really complex work, and actually I think only web browsers today do it. I'm not sure even operating systems do, because usually they don't operate with the same challenges that web browsers do where you have total UI flexibility, or as many people using the platform as web developers.

But it's also the only platform that has this architecture.

So if you're building desktop software, you can try to emulate this architecture and end up with some fairly good performance. Even just using nested frame buffers gets you really far. But if you're writing front-ends on the web, you're stuck between the browser's compositor or rejecting it entirely and writing your own stack in a graphics layer that gets pushed to a canvas.

None of this is good, it's all trade-offs, and I suspect the state of affairs will continue for years, because we don't have enough people doing compositing work. It's exceptionally specialized work.

As an addendum, the DOM isn't slow. This is just parroting. It's total crap. People who say this have no idea what they're talking about. What's slow is everything GPU related. That's not even the CSSOM, either. It's specifically the part where pixels get rasterized. But you don't have lots of people who work on compositors telling you that. You have a bunch of speculators thinking the DOM is slow without actually measuring anything.

Painting and drawing to backing layers is super expensive. Pixel fill is expensive. Changing document subtree node properties is not.

You absolutely need efficient invalidation in UI drawing. I have no idea what this author is talking about.

It's weird that the article mentions Makepad, but somehow fails to observe that Makepad's primary demo application is actually a software synthesizer. Go watch some of Rik Arends' talks! Most recently this one from RustNL last week: https://www.youtube.com/watch?v=9Q4yNlbfiYk (first talk of the day).

GTK 4, which the author mentions in their post, uses a retained rendering model so that the UI can be rendered on GPU. Widgets snapshot render nodes which are immutable and can even be cached between frames. Those render nodes are diffed to calculate the damage region automatically.

That damage is then used to scissor clip in GL before the render tree is converted to shader commands. The original damage flows through to eglSwapBuffersWithDamage() to limit the screen damage to the diffed areas. Some work is needed to do tiled/threaded processing of the render tree to reduce pathological cases instead of large scissor clips. Given how much a DAW changes per-frame when animating, it's probably close to pathological anyway.

In git, there is support for something akin to swap-chains so that you can use DMA-BUF textures (or similar, such as IOSurface), and specify the damage between each buffer. This is useful for things like virtual machines or other external plumbing where you have a buffer+damage to integrate into GTK's composition pipeline and don't want full-frame renders to result due to gsk_render_node_diff() falling over.

> Much of that finesse has been lost in this new generation of GUIs built on 3D APIs where the entire window is thrown away and redrawn every frame.

GTK 4 uses eglSwapBuffersWithDamage() scissor clipped to the union of the damage rectangles. Operations falling outside that clip are culled.

Only the damage areas are composited, assuming a competent compositor.