I once wanted to do something similar, so I wrote my own realtime audio streaming tool [0]; the UX resembles something more akin to a foot-gun than the polished (and much more feature-complete) Roc, but it does have excellent latency properties by running at the absolute ragged edge of what it can.
You could connect multiple clients to stream music to a single server (or multiple servers if you wanted to); the server kept a list of pending audio buffers for each client, mixing them all together into a ring buffer that gets spat out to portaudio. If a client underran, it simply missed that ring buffer rotation, and it "fell behind" by one buffer length (we request the minimum latency from portaudio, so this is usually measured in single-digit milliseconds). That would cause a few crackles and pops in the first second or two as the natural jitter of the network caused the client to underrun a few times, but then it would stabilize. (This didn't bother me much, as I usually was playing silence when I first connected it in any case).
In my experiments, the overall system latency was pretty close to the perceptual limit, I would estimate around 10ms, streaming over wifi from my laptop to a raspberry pi.
[0] https://github.com/staticfloat/popuset