Earlier quoted context omitted.
Instead of sandboxing the entire app, you should use privilege separation inside VLC. Low-risk code would run in processes with higher privileges, while the high-risk code (e.g. the decoders) would run in highly sandboxed processes and communicate with the privileged processes over a simple interface. This is the approach used by security-conscious server apps (e.g. openssh, postfix, dovecot) as well as Google Chrome…
It's a good question. But hard. > Does VLC currently have pretty clear separation between its various components? Very clear separation. One of the best, tbh. > Do you think it be much work to spin security-conscious parts like the decoders into separate processes? Extremely difficult. We've thought about it. For a video player the 3 parts that are sensitive, are protocols (file, http), demuxers (mkv, avi) and decode…
I was afraid that this was the case. Might shared memory be a viable solution to buffer sharing?
In any case, it makes me really happy to know that you've thought about this!