Maybe keep the Kurento project on the radar [1]. It provides you with "building blocks" to create an application, I think that's what you mean with
programmable: you use it to
build the application you need.
There are blocks for WebRTC, RTP, RTSP input (not output), and others. You can also apply computer vision to the video, with OpenCV filters. There is a "Composite" filter that places N streams in a grid. And users are expected to write their own fancy "blocks" if needed, so the option is there too.
Only important matter would be to see if the currently existing blocks are already providing the features you need, or new ones would have to be created.
Some technology from the Kurento project was acquired by Twillio [2], so it's possible (but not sure, I don't really know, maybe the parent commenter knows) that their media services might be based or inspired on Kurento.
Full disclosure I currently work for the Kurento project as the main developer, at Universidad Rey Juan Carlos.
[1]: https://www.kurento.org/
[2]: https://www.kurento.org/blog/whats-next-kurento-and-elasticr...
EDIT -- I wrote "under the radar" which is exactly the opposite of what I meant to say :-) now fixed