Self-hosting a high-availability Postgres cluster on Kubernetes

People use complexity as a boogieman to justify throwing together their own really wild chaotic & only so-so tested "simple" alternatives all the time.

To me, this feels like a modern wonder. We have layers of responsibility. Many people operate Kubernetes clusters already for all kinds of reasons. It provides a powerful broad base. Now on top of that, we can operate Postgres, with a very smart failover system that's super well tested & broadly used, that leverages this competent starting place.

What are the Fears Uncertainties and Doubts you have that make you scared about composition? What would help address specific concerns? To me, this division of responsibilities & use of consistent platform for a variety of needs feels like a huge win.

People love "simple" options but they're not. Run naked through the woods like savages option has appeal, but just getting started keeps adding up:

Sure, just add some bash scripts for some wal backups that go off-site. Easy! Install pgbouncer like one does, just a quick install, point it at the right systems. Setup some replication. Install and configure more kind of hairy software to make it HA. Configure some TLS cert yourself to not send naked traffic over wire. Add monitoring!

Then operationally, how quickly do you think you'll be able to fail over (with ogbouncer staying ok), failback, do an upgrade, add more replicas, lose a replica? Can you rotate cert reliably in a timely fashion? Can your replacement? How well did you document everything? Will you have all the monitoring you need when incidents start coming in, or did you just spitball a couple metrics into place?

The alternatives, in my view, are obviously bad. You can do them. Either cheaply with risk or industriously with effort & applied-talent. But having cohesive autonomic systems at our back that try to help, that can faultlessly do many common tasks with perfect accuracy (across unimaginable numbers of systems, with perfect consistency, in record time): that feels like a massively better place in the universe, one that I don't get why so many people kick scream & drag against. Rarely are their arguments well elaborated ("scary"), and their counter-suggestions feel like they massively underrated how multi-faceted & carefully connected production systems are, for good reason, and how hard it can be to remember to not forget to change X when you do Y.

That being said, we're also layering a bunch of stuff on top - helm, nginx, GKE, terraform, as well as a mountain of other things, and then to top it off we have a bunch of shell scripts doing random things to help tie it all together.

Normally I can pick things up pretty quickly. I just built a parser with tree-sitter, despite knowing virtually nothing about language design. Didn't take very long.

But the modern devops stack is a learning curve like I've never seen before. It's taking me more energy to learn it than it did for me to learn programming itself. Then again, maybe I'm just getting old.