Opus is specifically designed to be low latency. I'm not sure where you got the idea that latency would be a problem. Of all the lossy codecs, there isn't much that is faster.