Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-hosting a high-availability Postgres cluster on Kubernetes

ryan-schachte.com

Post reply on HN
Print view View original

11–20 of 92 posts

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#11

Post by williamdclt »

renegade-otter wrote: post #5
gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

This kind of resilience is a form of art, and it's also kind of a full-time job. I would not advise trying this for a "side project at work". Generally we all agree that we move to the cloud and it's "fully managed". If it goes down - that's the price we pay. If you have to ask the question "what if the RDS goes down", then you are really in a different universe. That last guarantee of uptime requires a ton of work,…

> If you have to ask the question "what if the RDS goes down", then you are really in a different universe.

It does go down though, don’t neglect the possibility because it likely will happen. With very average workloads, I’ve seen RDS databases restart unexpectedly, read replicas being completely out of service, and even databases being completely frozen (can’t even connect as root).

I’d still go with managed, but it certainly doesn’t give full reliability :) you still have to consider “what if it goes down” - it will!

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#14

Post by candiddevmike »

renegade-otter wrote: post #5
gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

This kind of resilience is a form of art, and it's also kind of a full-time job. I would not advise trying this for a "side project at work". Generally we all agree that we move to the cloud and it's "fully managed". If it goes down - that's the price we pay. If you have to ask the question "what if the RDS goes down", then you are really in a different universe. That last guarantee of uptime requires a ton of work,…

> That last guarantee of uptime requires a ton of work, testing, and money, because you are all the way up and to the right on the curve of diminishing returns.

Not when it's a core part of your business...?

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#15

Post by _joel »

ssijak wrote: post #10

Is Kubernetes still hard in 2024?

I don't know about hard but it's fairly straightforward to spin up clusters and maintenance seems to have become less of a headache (at least imhe). It depends on what you will be doing with the cluster and how you use it.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#17

Post by levkk »

gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#18

Post by jamesu »

ssijak wrote: post #10

Is Kubernetes still hard in 2024?

From recent experience I'd say it's the sort of tech that starts off simple enough with the right distribution, but then gets more complicated the deeper down you dive.

Probably the most hard thing I found was wrapping my head around the way the storage works.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#19

Post by turtles3 »

andrewmunsell wrote: post #7
x86hacker1010 wrote: post #3

Earlier quoted context omitted.

Any reason you landed on that Operator compared to what OP is using (Zalando)?

Honestly no, it's mostly due to inexperience with operators and not really understanding what the "best" way to find operators is. I did also look at the Crunch Data one (I was having some issues setting that one up), but didn't even find Zalando during my search. OperatorHub is currently the main resource I use, but GitHub stars aren't exposed in the search so I have been looking at the "Capability Level" chart and…

A subtle advantage of cnpg is that it doesn't use statefulsets, instead the operator handles things like mapping storage volumes and stable identities. Regular kubernetes statefulsets have some tricky sharp edges for failure recovery.

I don't know if all of these alternatives use statefulsets but I remember several doing so.

I've personally found cnpg to be pretty robust, and supports everything you will eventually need once you're locked into a solution (eg. Robust backups, CDC, replica clusters).

I'm yet to find anything of a similar standard for mysql.

EDIT: it should also be noted that CrunchyData is a proprietary solution and requires a license to use in production. This is not particularly obvious from their docs.

Post reply on HN