Earlier quoted context omitted.
The "user is the only security boundary" ship sailed long ago with, chroot, SELinux, AppArmor, Snap, Flatpak, namespacing. It will continue to be a bumpy ride retrofitting an ecosystem not made for the tighter boundaries but it's still the goal.
Many of those work by running processes under what is effectively a subuser. The problem with it is, that it works fine when one purely speak of being able to write and read from files, but the moment servers such as display servers or Pulsaudio and DBus come into play, the picture becomes more difficult. All of those technologies work on a simple binary level where the subuser has access to the socket, or it does no…
Pulseaudio is not getting much work these days, I believe the work currently is happening in Pipewire, which was built to have a fine-grained permission system that can work with any sandbox and is backwards-compatible with Pulseaudio.
I'm not sure how your IRC bouncer would work there, but presumably if you sandboxed that, it only needs to talk to the IRC socket and nothing else. For other obscure servers that have no concept of security, I'm not sure what can be done about that if nobody wants to modify them or replace them. You might just have to accept that they will need to run at elevated privileges and can clobber your system or home directory.