Earlier quoted context omitted.
Any reason for doing audio and video work on the backend rather than adopting the (much easier to scale) SFU model and just forwarding packets? Similarly for the physics, seems like doing that work frontend would scale much better.
Even if you use stream forwarding only, network will be a major O(n) cost factor when using cloud providers, at least for video. There is also some super interesting middle ground between full video forwarding and selective reencoding by using SVC for cheap video resizing on the backend.
Bandwidth will be a high cost no matter what (and anyone planning to scale up real-time video better be prepared to move off the cloud at some point) but needing compute resources for encode/decode on the backend makes large scale realtime video infeasible for any reasonable cost. The only successful large scale deployments push most compute work to the clients.
An extra decode/encode step also adds latency and the total latency budget before UX is impacted is small.