Native Matrix VoIP with Element Call • Hacker News Board

Re: Native Matrix VoIP with Element Call

#41

Post by Arathorn » Sat, Mar 05, 2022, 5:12 PM UTC

jeroenhd wrote: post #16

I wonder if this setup can be leveraged to create a Twitch-like streaming solution based on Matrix. The group call setup is clearly not built for such a use case, but if you only stream to a small audience (say, people paying for a certain perk on Patreon?) you could probably use this quite easily. The client would need better support for alternative inputs, of course, such as the RTMP suggestion found on the Github…

> I wonder if this setup can be leveraged to create a Twitch-like streaming solution based on Matrix.

Yes, we've built it with this in mind. There are three possible approaches:

* You can broadcast a headless client via HLS or RTMP, similar to how Jibri works for Jitsi, and how we broadcast FOSDEM (https://matrix.org/blog/2021/02/15/how-we-hosted-fosdem-2021...). Basically you run a headless Chrome against a virtual X server and pipe X's framebuffer into ffmpeg. It's a pretty blunt approach and obviously uses a lot of serverside resources, but ensures that the broadcasted stream is completely faithful given that it's literally a recording of what a client would be seeing.

* Alternatively, you could do the same approach, but pipe it into a WebRTC broadcasting platform rather than HLS/RTMP/RTSP/DASH. This could give much lower latency, but more bandwidth on the server given you're effectively setting up a 1:1 VoIP call with each viewer, and you'll be unlikely to be able to use CDNs (unless we end up in some crazy world where freeform multicast on the internet finally comes to pass thanks to CDNs fanning out your RTP packets for you :P)

* Finally, you could composite the streams together serverside using an MSC3401 compliant MCU and broadcast the result via one or other approach. This might not be quite as high fidelity as running a 'real' Element Call instance serverside, but could be way more efficient in terms of resources, and could prove perfectly adequate. The only catch is that we haven't bolted MSC3401 onto any MCUs yet. (Clearly we should get back in touch with our friends at FreeSWITCH / Signalwire :D)

> What I'm also curious about is how this fits in with the experiments on the peer-to-peer mesh system (running a homeserver on every client). Forwarding video across a web of clients would become bandwidth and CPU intensive real fast and the added latency would be non-negligible.

MSC3401 is pretty much orthogonal to P2P Matrix. The way it works is that it'll use whatever conferencing nodes ('foci') are advertised in the room to mix together the calls in a decentralised fashion. If there are none (as per today) then it just goes full mesh. If there is one, then everyone will converge on it. If there are two, then folks will pick the one with the lowest latency. But critically, it doesn't matter whether the clients are talking to a serverside homeserver or are running P2P Matrix. Finally, the foci could run either serverside or clientside (much as skype supernodes were effectively clientside foci), but we need to get serverside foci working first before we add clientside ones in for the P2P crew ;)

Re: Native Matrix VoIP with Element Call

#42

Post by zaik » Sat, Mar 05, 2022, 5:12 PM UTC

GeckoEidechse wrote: post #29

encryptluks2 wrote: post #19

I feel if XMPP just had some better tools this push for Matrix would be unnecessary.

https://matrix.org/faq/#what-is-the-difference-between-matri...

If you read https://github.com/matrix-org/matrix-spec-proposals/blob/mat... it becomes apparent that the 'eventually consistent' part of Matrix is more of a hindrance in this case.

Also 'the more federation and interoperability the better' is kind of contradictory to constantly reinventing extisting Internet Standards. I suspect VoIP over Matrix is not compatible with SIP or XMPP A/V calls and looking at current state of the matrix.org XMPP chat implementation they probably will never be.

Re: Native Matrix VoIP with Element Call

#43

Post by georgyo » Sat, Mar 05, 2022, 5:13 PM UTC

jeroenhd wrote: post #12

youerbt wrote: post #8

> And, the big one drum roll, please ... we will be integrating this into Element so you can have voice and video rooms, and hold group video calls inside the Element app natively over Matrix. I'd love to replace my teamspeak server with this. Discord is tempting, but I'd rather host it myself.

For what it's worth, Matrix / Element already have something like this with the built-in Jitsi integration in (some) clients. The UX isn't great right now in my opinion (which will hopefully change when this project gets merged into Element) but if all you want is rooms, text and voice, then you're already set with the current setup. You can self-host a Jitsi instance and use it in Matrix rooms as an integration. Dep…

The UX is more than terrible, it is basically unusable.

The jitsi stuff is bolted on, you join a link and your user name does not come with you.

No discoverablity. Does this channel already have a jitsi room attached, no way to know. I see a link to a jitsi room, is there anyone inside? No way to know.

Recommending this as a alternative to TeamSpeak, mumble, or discord today is a way to get people to try a horrible experience and then say Matrix sucks forever.

Only when this call stuff merges will Matrix start being viable for those communities.

Re: Native Matrix VoIP with Element Call

#44

Post by jfkimmes » Sat, Mar 05, 2022, 5:13 PM UTC

Arathorn wrote: post #20

From my perspective, the really exciting thing about this that it works equally well in mobile web browsers as well as desktop web - clicking on a link on Mobile Safari should Do The Right Thing without having to install anything. Moreover, because it's built on Matrix, MSC3401 ( https://github.com/matrix-org/matrix-doc/blob/matthew/group-... ) means that we'll finally have decentralised cascading video/voice confere…

Could you give some insights on how you estimate the amount of effort of making Element Call competitive, performance-wise, with say Discord. I heard that Discord threw a lot of time and money at optimizing voice in their product. Can you just jump in and realistically compete?

There are a lot of performance/latency/sound quality comparisons online of Mumble vs TeamSpeak vs Discord and recently Jitsi vs MS Teams vs Zoom, etc. I feel like this is a problem-space that can be optimized to an arbitrarily deep extent. Just two examples that come to mind are SFU performance/efficiency and noise reduction, two things where e.g Jitsi notoriously lags behind Zoom.

Re: Native Matrix VoIP with Element Call

#45

Post by Arathorn » Sat, Mar 05, 2022, 5:14 PM UTC

panick21_ wrote: post #39

Arathorn wrote: post #33

Earlier quoted context omitted.

MSC3401 specifically provides for this UX, with the `m.intent` field used when you instantiate an `m.call` within a room ( https://github.com/matrix-org/matrix-spec-proposals/blob/mat... ): * `m.intent` to describe the intended UX for handling the call. One of: 1. `m.ring` if the call is meant to cause the room participants devices to ring (e.g. 1:1 call or group call) 2. `m.prompt` is the call should be presented as…

So will the room have a setting to make a voice video only room, a mixed room or a chat only room? I assume m.ring would not be needed if its a something like a Teamspeak room.

It wouldn't be a setting; the intent would be set based on how you initiated the call. If you hit a big 'call Alice and Bob!' button you'd get a call which rings them both; if you hit a 'start a conference' you'd get a Teams/Zoom-style conference; if you hit 'create a voice/video room' button you'd get a Discord-style voice/video room.

(N.B. we might not bother with the group-calling option in Element; haven't decided yet - but Matrix needs to support it for folks who want those semantics :)

Re: Native Matrix VoIP with Element Call

#46

Post by Arathorn » Sat, Mar 05, 2022, 5:17 PM UTC

panick21_ wrote: post #37

This is really cool. I really feel like this ecosystem is hitting its stride. The list of upcoming features is long and sometimes I don't understand how they can have so many things going on at once. Features tend to take a few years to mature, but that is not a bad thing. I will be setting up my own server shortly, something that I have been thinking about for a while and maybe try to develop something. Congrats to…

> The OpenID part, would it not be cool if Matrix would offer something like a federated SSO experience?

Yes, hopefully we'll be able to use the OIDC server bundled with your Matrix server to auth users in general if wanted (or alternatively you can of course point at an existing OIDC server).

Re: Native Matrix VoIP with Element Call

#47

Post by kitkat_new » Sat, Mar 05, 2022, 5:17 PM UTC

panick21_ wrote: post #39

Arathorn wrote: post #33

Earlier quoted context omitted.

MSC3401 specifically provides for this UX, with the `m.intent` field used when you instantiate an `m.call` within a room ( https://github.com/matrix-org/matrix-spec-proposals/blob/mat... ): * `m.intent` to describe the intended UX for handling the call. One of: 1. `m.ring` if the call is meant to cause the room participants devices to ring (e.g. 1:1 call or group call) 2. `m.prompt` is the call should be presented as…

So will the room have a setting to make a voice video only room, a mixed room or a chat only room? I assume m.ring would not be needed if its a something like a Teamspeak room.

you can choose one of the three, and if you choose option 3, the room will become a voice video only room (this is how I understand it).

So m.ring existing isn't a problem.

How it is exposed in the UI, is a completely different thing, although I don't think it would be impossible to make a button that creates a room with a m.room call aka voice/video room

Re: Native Matrix VoIP with Element Call

#48

Post by Arathorn » Sat, Mar 05, 2022, 5:20 PM UTC

arianvanp wrote: post #31

jfkimmes wrote: post #26

Earlier quoted context omitted.

The post says there is no E2EE at the moment. They're probably talking about signalling, since with the current implementation of full mesh you get E2EE 'for free' in WebRTC. Also the post mentions: https://2021.commcon.xyz/talks/extending-matrix-s-e2ee-calls...

Mesh doesn't scale for large conference calls though. Usually some SFU needs to be in play. But that's only possible if you have Insertable Streams support which for now only Chrome has

Safari DP 141 actually released RTCRtpScriptTransform support 2 days ago (thanks to https://trac.webkit.org/changeset/270107/webkit/). But we're waiting eagerly for Firefox to add it in https://bugzilla.mozilla.org/show_bug.cgi?id=1631263.

Hopefully by the time we've sorted out the SFU component, all the browsers will have RTCRtpScriptTransform implemented and so we'll get good cross-platform E2EE for larger calls (rather than it being limited to Chrome + Safari + Desktop)

Re: Native Matrix VoIP with Element Call

#49

Post by mawise » Sat, Mar 05, 2022, 5:33 PM UTC

Arathorn wrote: post #14

jfkimmes wrote: post #11

Earlier quoted context omitted.

I feel like this is a misunderstanding. You can already self host 'Element Call' and point that to your own homeserver. The passage you're referencing just refers to their hosted Element Call client. I'm pretty sure they're currently trying to move away from credentials-based login on 3rd-party clients to token-based authentication. I would guess they open up their hosted client when they finished implementing this n…

Yup, the grandparent post is a misunderstanding. Basically, we want Element Call to work with a single click to a URL without requiring any account - much as Jitsi and similar do. Therefore it needs to pick a default homeserver. Currently it doesn't play well with existing Matrix accounts (as it will sync in loads of chatrooms which are irrelevant when you just want to do a call), so for expedience we just create acc…

I guess I still don't understand it. (And my apologies for coming across like I was trying to pick a fight)

What does it mean that I can currently use it with my existing homeserver? The architectures for some of these federated systems can be difficult to understand, I feel like I didn't really understand what _Matrix_ was until I tried running my own homeserver.

Re: Native Matrix VoIP with Element Call

#50

Post by Arathorn » Sat, Mar 05, 2022, 5:56 PM UTC

jfkimmes wrote: post #44

Arathorn wrote: post #20

From my perspective, the really exciting thing about this that it works equally well in mobile web browsers as well as desktop web - clicking on a link on Mobile Safari should Do The Right Thing without having to install anything. Moreover, because it's built on Matrix, MSC3401 ( https://github.com/matrix-org/matrix-doc/blob/matthew/group-... ) means that we'll finally have decentralised cascading video/voice confere…

Could you give some insights on how you estimate the amount of effort of making Element Call competitive, performance-wise, with say Discord. I heard that Discord threw a lot of time and money at optimizing voice in their product. Can you just jump in and realistically compete? There are a lot of performance/latency/sound quality comparisons online of Mumble vs TeamSpeak vs Discord and recently Jitsi vs MS Teams vs Z…

The competitive gap with Discord in terms of media quality is probably something like:

* Need a low-latency SFU. This should be very doable; not only are there a lot of good FOSS SFUs to build on top of these days, the history of the Matrix team is actually that we built VoIP stacks fulltime before we shifted focus to Matrix, and we've built MCUs and media servers of all flavours in the past. MSC3401 should also give us a competitive edge given latency will be automagically minimised by using the physically closest decentralised SFU, thus letting anyone bring their SFU to the party.

* Needs a SFU with good rate control (and/or FEC). This is probably the single most important thing to get right in terms of quality. Signal wrote up a good overview of why: https://signal.org/blog/how-to-build-encrypted-group-calls/

* Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc). Ideally you need something like https://krisp.ai/ or https://workspaceupdates.googleblog.com/2021/06/background-n... in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard. However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (https://github.com/matrix-org/seshat), we'll have a go at doing something similar for this problem too.

* Excellent automatic gain control. The importance of normalising/compressing everyone's audio so they're equivalent loudness is really important.

We're also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds.

We can probably also be more bullish about supporting new audio codecs like Lyra.

EDIT: oh, the other obvious thing is echo cancellation - but we're currently at the mercy of the browser's WebRTC stack for that. However, we could ship a tweaked WebRTC in Element Desktop (or other native Matrix clients) in future to do better than plain old WebRTC.