Earlier quoted context omitted.
Can the clients not advise the server what bandwidth and packet loss rate they are seeing, so the server can adjust their traffic rate? There is no reason not to allow a signalling channel separate to the E2E encrypted data channels.
The server would then have to tell the streamer / talker to reduce their quality for that one client while degrading all other clients that have a fine connection right?
If you wanted to be fancy you could do distributed processing, whereby another node downscales and re-publishes the feed.