Digital Audio Workstation Front End Development Struggles

> text rendering is definitely one of the things that come up the most

Ok this doesn’t dispute what I’m saying at all and I would think you would know better. Do you have profiles from 15 or so years ago that show text rendering was a substantially smaller percentage of time.

And if you do - ensure it is comparing apples to apples with subpixel (or at least greyscale) hinting and full featured shaping. Neither of these are related to blitting/blending performance but they are relatively expensive.

Modern text rendering is super expensive (but by modern we’re talking on the order of 15 to 20 years, not yesterday). People are making claims that CPU 2D rendering has gotten more sluggish in that time frame - though nothing fundamental has changed other than the prevalence of “Retina” in some circles - but that isn’t something that has scaled past CPU performance. This all works because historically these systems have been painstaking about only drawing when needed what’s needed.

If anything has changed it’s that - software in general for a number of reasons has gotten much sloppier about this.

And in the case of Mac OS specifically - they continue to throw all kinds of shit in the CPU rendering pipeline. If you just do things yourself and blit to a Metal texture, nothing is fundamentally slower.

I contend that most of what people are claiming are some fundamental hardware changes (that haven’t happened - things are faster across the board but otherwise it’s the same basic stuff) is due to less effective application software methods. For 2D desktop apps the GPU is not the factor some people seem to think it is.

Some comments on the issues mentioned at the beginning of the article as I had the same questions in https://ossia.io:

- DAWs have a lot of toolbars and panels (browser panel, timeline panel, piano roll panel, fx rack panel, mixer panel, audio editor panel, automation editor panel, settings panel, etc.).

for this Qt always performed brillantly for me. I went from QDockWidget to QSplitter-based layout though.

- Some widgets like decibel meters and other visualizers are constantly being animated, meaning the GUI library needs to efficiently redraw the screen every frame.

all the CPU-based toolkits will only redraw what changed though

- In addition, visualizers can be expensive to render on the CPU (especially spectrograms/spectrometers). Ideally you should use custom shaders to render them on the GPU.

fair, but then you may pay the cost of a GPU -> CPU transfer which isn't always free

- Clips on the timeline are notoriously expensive to render. There needs to be some way to cache the contents of clips into a texture (Either directly or by making use of the GUI library's "damage tracking" which I'll get into later.) Audio clips are the biggest culprit, because rendering waveforms requires the CPU to first do a linear search through the source material for peak values, and then render the waveform pixel-by-pixel (or even better use custom shaders to send commands to the GPU).

AFAIK a lot of DAWs perform a full scan when the file is loaded and save the result in a database or in a file next to it (e.g. Reaper's .reapeaks, Ableton's .asd...) so that you don't need to perform a complete rescan. In ossia.io I use three different algorithms / display methods depending on the zoom level used: at "far away" zoom it uses the minmax of the audio slices, at intermediary zoom it draws lines and when getting closer, it starts drawing individual samples.

https://github.com/ossia/score/blob/master/src/plugins/score...

- Automation clips can contain a bunch of bezier curves, which are slow to render.

Convert those to line segments with an approximation setting that looks good enough and it'll be ten times faster (keep the bezier for your data model of course).

https://github.com/ossia/score/blob/master/src/plugins/score...

https://github.com/ossia/score/blob/master/src/plugins/score...

- Piano roll clips can contain lots of little rectangles in order to display a "minimap" of the MIDI notes inside of it.

oh damn yes, I spent so much time on this and it still needs so much optimizing... if someone wants to give a shot at it :D

https://github.com/ossia/score/blob/master/src/plugins/score...

https://github.com/ossia/score/blob/master/src/plugins/score...

- On top of all this, clips can contain text labels which can also be expensive to render.

Yep, made myself a few "cached text" Qt items over time as the builtin cache wasn't satisfactory

https://github.com/ossia/score/blob/master/src/lib/score/gra...

- The fact that a timeline is zoom-able also makes it harder to cache the rendering of clips. If the timeline changed its zoom level, all visible clips pretty much have to redraw all of their contents.

yep

- Piano rolls can also be expensive to render if there is a bunch of MIDI notes, especially if there are text labels on the notes.

yep

- If the user clicks on a folder in a sample browser containing hundreds or even thousands of files, allocating a label widget for each file in the browser list will be very expensive. Something like the list factory in GTK is needed here.

yep, Qt's also able to cache this. Though for instance for Qt's QFileSystemModel I carry a small patch to disable any kind of sorting when there's more than a few hundred thousand files (which happens for large media libraries): https://github.com/jcelerier/qtbase/commit/9909c3c7902cf7a2b... and also worked a bit with upstream Qt to get it to improve (for instance it was regenerating regexes ALL THE TIME when filtering for specific file extensions)

- We want to reserve as much CPU as possible for the actual audio processing. Ideally the GUI shouldn't take up more than one or two CPU threads. - On some platforms, we also need to make sure there's actually enough CPU left for 3rd-party plugins to render their GUIs.

yep

Absolutely, such a great project! I understand one of the guys behind the project is doing it more or less out of passion as he made his money from WinAmp back in the day?

and without GPUs, mind you. And a lot of that trickery probably came down to: let's write everything in assembly. Fruity Loops was absurdly fast in the early 2000s and was written in Delphi and assembly, IIRC.

In 2023 DAWs should not be that difficult to make fast. It's a solved problem. Multi-channel audio processing is one thing that modern CPUs should not even break a sweat at. I believe this article has uncovered the basics that have made every DAW complex from the beginning, but seems to be focused on efficiency with your bog standard off-the-shell GUI library. That's the mistake, IMO. None of the DAWs I know of use, for example, standard GTK. Even the ones that do use a GUI toolkit use it minimally, for the file pickers and configuration. Not as the primary interface for the DAW itself. You need to go lower level and remove the layers of cruft. I mean hell, adding language bindings on top of everything is going to add yet another layer of inefficiency.

VDOM has nothing to do with garbage collection. VDOM is a cache.