Live data from Hacker News

Modern messaging: Running your own XMPP server

codedge.de

71–80 of 156 posts

Re: Modern messaging: Running your own XMPP server

#71
post #55

If you're looking for something that's a little less hassle and has some very sane defaults, try https://snikket.org/

Has anyone done a security review of their source code?

It is simply Prosody + Conversations + Siskin [1], so I'd say that many people have had their eyes on their code.

Specific security audits would have to be searched for, though.

[1]: https://snikket.org/open-source/

Re: Modern messaging: Running your own XMPP server

#72
post #14

Last time I tried this I couldn’t even find an iOS client that supported reactions. Hate on the complexity of Matrix all you like, but at least when you ask your family to use it, they get a modern experience and don’t feel like they’re giving up everything vs every other messenger they are used to.

XMPP lacked good, modern clients, so folks designed a terrible protocol, and then made modern clients with this kind of feature for that protocol.

There’s an extra step that could have been skipped entirely there and we’d be better off.

Re: Modern messaging: Running your own XMPP server

#73
post #64
post #14

Last time I tried this I couldn’t even find an iOS client that supported reactions. Hate on the complexity of Matrix all you like, but at least when you ask your family to use it, they get a modern experience and don’t feel like they’re giving up everything vs every other messenger they are used to.

> Last time I tried this I couldn’t even find an iOS client that supported reactions. Hate on the complexity of Matrix all you like, but at least when you ask your family to use it, they get a modern experience and don’t feel like they’re giving up everything vs every other messenger they are used to. Now if only it actually worked consistently and reliably. I'm not being snarky, the app is flakey as hell. Reactions…

> Now if only it actually worked consistently and reliably.

It does. I use a self hosted Matrix server for chat with friends (multiple servers federated, even!) and it is rock solid.

Re: Modern messaging: Running your own XMPP server

#77
XMPP has very nice server implementations, and the protocol is OK regarding complexity.

But the clients are lacking. On linux there is gajim that is "okish" but it lacks calling capacity with mobile clients. On mobile there is conversation and derivatives on android which are "nearly there" and monad on iOS.

Globally, the main lacking features are:

- voice and video calls cross platform

- gif integration, tenor/giphy/imgur...

- fast sync (if you open gajim after being offline for a week, it takes ages to catch up)

Re: Modern messaging: Running your own XMPP server

#78
post #14

Last time I tried this I couldn’t even find an iOS client that supported reactions. Hate on the complexity of Matrix all you like, but at least when you ask your family to use it, they get a modern experience and don’t feel like they’re giving up everything vs every other messenger they are used to.

There isn't a single XMPP client for iOS that properly follows the iOS HIG and isn't ugly or unusable. There are very few clients that are even fully featured. The XMPP landscape on macOS is even more bleak.

Isn’t this true of most open source software in iOS?

The intersection of developers who want to develop on and for a proprietary walled platform and also want to work on open source clients for open standard, descentralised protocol must be pretty small.

Re: Modern messaging: Running your own XMPP server

#79
post #17

One potential issue with XMPP is the default port is commonly blocked on public wifi. There is an nginx-xmpp to proxy it, but it is archived. https://github.com/robn/nginx-xmpp For that reason http based protocol just seems much easier on the network or something that can be easily reverse proxied without extensions will be easier to self-host and have wilder internet connection accessibility.

The default ports are often blocked on such networks (public wifi, corporate firewalls, etc.), but also often open. 5222 is used by e.g. WhatsApp, 5223 is used by e.g. Apple push notifications. So it's not as bad as it could be. But it's also totally possible to run XMPP over 443, and this is a feature of many popular XMPP deployments. If you're self-hosting, there are some guides for different deployment approaches…

> 5222 is used by e.g. WhatsApp

I left in 2019, but when I was there, WhatsApp used port 5222, but the client would try port 443 if port 5222 didn't work. After it had tried those enough, it would try on port 80 with HTTP wrappers.

Really, the right model for a public service is what AOL did for AIM. Listen on all the ports. Clients should try on the 'proper' port, then 443, then 80, then random permutation. Skip certain ports because nobody likes it if you probe on smtp, smb, or chargen (etc)

Re: Modern messaging: Running your own XMPP server

#80

Earlier quoted context omitted.

You're thinking of outbound port limits. Inbound connections all come in on the same port and there's no port-related limits there. The real limits are going to be on hardware. If we want 40M concurrent connections, the server will need a few hundred to a few thousand gigabytes of memory.

Unless you're doing multicast or anycast, there's a port bound IP handshake that happens. You have your listener (your server port) and the connected TCP/IP client (client port, on the server machine). You're limited to 64533 clients (0-65535 but the first 1000 are reserved). If there's a way to get more client on a single machine, I'm all ears - but that's baked into IP protocol. TCP/UDP doesn't matter. It's a limit…

You should be able to deal with 4 million clients on one server in 2025; we did 2 million on one chat server in 2012 [1]. I can find documentation of 2.8M in Rick Reed's Erlang Factory presentation [2], page 16. That was part of a clustered system with chat connections on one group of servers and several other groups of servers for account information, offline messages, etc. Also, the connections weren't Noise encrypted then, and there were a lot more clients in 2012 that didn't have platform push, so they would try to always be long connected ... it's not very hard to manage clients that just need to be pinged once every 10 minutes or so.

But this was with only 96G of ram, and dual westmere 6-core processors. You can put together a small desktop with consumer parts that will be way more capable today. And if you run everything on a single machine, you don't have to deal with distributed system stuff.

When I left in 2019, we were running on Xeon-D type hardware, with maybe 64 GB ram and IIRC, only doing about 300k connections per chat machine. Chat scales horizontally pretty well, so run a couple machines of whatever, find your bottlenecks and then scale up the machines until you hit the point where it costs more to get 2x of bottleneck on a single machine than to get 2 machines with 1x of the bottleneck. I suspect, if you can get quality AM5 servers, that's probably the way to go for chat; otherwise likely a single server socket would be best; dual socket probably doesn't make $/perf sense like it did 10 years ago. If you get fancy NICs, you might be able to offload TLS and save some CPU, but CPU TLS acceleration is pretty good and there's not a ram bandwidth saving from NIC TLS like there is for a CDN use case.

IMHO, getting a single server to support 4M clients shouldn't be too hard, and it should be a lot of fun. Mostly all the stuff that was hard in 2012 should be easy now between upstream software improvements and the massive difference between CPUs in 2012 and 2025. The hard part is getting 4M clients to want to connect to your server. And trying to setup a test environment. Elsewhere on this thread, there's a like to the process one blog from 2016 where they ran 2 M clients on a single server (m4.10xlarge (40 vCPU, 160 GiB) with a single same spec server as the client load generator; the impressive part there is the client load generator --- I've always needed to have something like a 10:1 ratio of load generators to servers to load down a big server. And they managed to get that to work with all their NIC interrupts sent to a single cpu (which is not what I would have recommended, but maybe EC2 didn't have multiple nic queues in 2016?)

> If there's a way to get more client on a single machine, I'm all ears - but that's baked into IP protocol. TCP/UDP doesn't matter. It's a limitation of IP.

As others have said, the limitation is the 5-tuple: {protocol, SrcIp, DstIp, SrcPort, DstPort}; if you're a server, if you hold SrcIP and SrcPort fixed, and for each dest ip, you can have 64k connections. There are a lot of dest ips, so you can host a lot of connections, much more than you can actually manage on a single system. If your clients are behind an aggressive CGNAT, you can actually run into problems where there's more than 64k clients that want to connect to your single IP and port ... but you can listen on multiple ports and address it that way, and most aggressive CGNATs are only for v4; if you listen on v6, you'll usually see the client's real ips or at least a much more diverse group of NAT addresses.

If you listen on all the ports, you can have 4 billion connections between you and any given IP. That's not going to be limiting for a while.

[1] https://blog.whatsapp.com/1-million-is-so-2011

[2] https://www.erlang-factory.com/upload/presentations/558/efsf...

Post reply on HN