Earlier quoted context omitted.
I would've explored alternative servers. The issue with Synapse is the same as I currently have with Lemmy - lack of HA. One particular machine goes down and the system is 100% unavailable. And most of my machines are hardware I own, located at homes (mine, parents, etc) in different countries, so they're slightly more prone to random power or uplink outages than AWS' us-east-1. Hobbyist-grade geo-distributed cluster…
Solving active/backup HA for Matrix is pretty straightforward: postgres replication for the db and a shared filesystem for the media repo. When the primary postgres+synapse goes down, promote the secondary to primary, and continue. We’re also working on account portability (MSC4014) that would eventually support active/active at the matrix layer, but it’s not ready tet.
PostgreSQL replication is hard for me. I know how to set up "classic" replication, but automatic failover and recovery that can solve split-brain issues is, sadly, not something I've figured out. And if it's not the machine, but the network that goes down (which happens few times a year on consumer-grade connections), it's a problem that happens in practice. I have learned how to repair nodes by hand, but I don't want to do this.
So far, I've managed to avoid thinking much of it by using distributed databases (CockroachDB and Consul, in particular) that already have all the magic built-in, so I just have to be careful with the settings.
Maybe I just need to learn how to deal with PostgreSQL...