Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-hosting a high-availability Postgres cluster on Kubernetes

ryan-schachte.com

Post reply on HN
Print view View original

61–70 of 92 posts

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#61

Post by levkk »

DasIch wrote: post #49
levkk wrote: post #17

Earlier quoted context omitted.

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

Here is an announcement from last year stating that volume expansion is stable since 1.24: https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/ The zalando postgres-operator also mentions as a feature: > Live volume resize without pod restarts (AWS EBS, PVC)

What about changing iops or storage class?

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#62

Post by SlightlyLeftPad »

debarshri wrote: post #45
williamdclt wrote: post #11

Earlier quoted context omitted.

> If you have to ask the question "what if the RDS goes down", then you are really in a different universe. It does go down though, don’t neglect the possibility because it likely will happen. With very average workloads, I’ve seen RDS databases restart unexpectedly, read replicas being completely out of service, and even databases being completely frozen (can’t even connect as root). I’d still go with managed, but i…

Problem is that RDS comes at a price. It is purely about operation cost. When you have 1500+ databases these cost add up. At that point, this kind of techniques are required to self host the databases. Price per DB with HPA na VPA is way lower than what you would pay for managed databases as well as you can hire a full time devops+dbadmin and still be cheaper.

This is a perfect example of a cost savings opportunity that offers no benefit and is really just a foot gun when it comes to operational complexity. The cloud cost savings almost never justify the operational costs here. There’s typically a very long list of other cost optimizations to be made before this would ever be on the table for me. If you’re talking about 1500+ databases and think a single full time devops+dbadmin in some kind of unicorn person is going to be a better option, I feel bad for both of you.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#63

Post by gbartolini »

levkk wrote: post #17
gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

I don’t agree here. There are operators like the one I’m a maintainer of (CloudnativePG) which works directly with Kubernetes, teaching it how to handle Postgres clusters as a coordinated set of instances. Enormous improvements have been done in the last couple of years, and we are particularly focused in working together with storage groups in Kubernetes to handle database workloads, such as for example declarative support for tablespaces and volume snapshots.

I suggest you to read CloudnativePG documentation as well as this article: https://www.cncf.io/blog/2023/09/29/recommended-architecture...

Also watch the video of my talk at last Kubecon in Chicago about handling very large databases.

I hope this helps.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#65

Post by siamese_puff »

Havoc wrote: post #64

I wonder how much of big cloud one can replicate on a diy cluster. Database works via this. S3 via minio. Redis. And maybe openFAAS? That seems like a lot of the key building blocks already.

I'm biased, but I agree! This is a great article that further illustrates how we are a bit conditioned to use off-the-shelf vendor tooling that we can run ourselves.

https://kiwiziti.com/~matt/wireguard/

Of course, there are tradeoffs you have to make (security, uptime, criticality of the system), but homelabs exist as perfect experimentation frameworks.

At $DAYJOB we run a global scale Ceph cluster (S3-like API/object storage), so even at a large scale it's not impossible to imagine.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#66

Post by moondev »

levkk wrote: post #61
DasIch wrote: post #49

Earlier quoted context omitted.

Here is an announcement from last year stating that volume expansion is stable since 1.24: https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/ The zalando postgres-operator also mentions as a feature: > Live volume resize without pod restarts (AWS EBS, PVC)

What about changing iops or storage class?

https://kubernetes.io/docs/concepts/storage/volume-pvc-datas...

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#67

Post by peterbecich »

andrewmunsell wrote: post #2

My holiday project was doing another pass at my Homelab Kubernetes cluster, part of which involved switching to a proper operator to manage Postgres. Coincidentally, I setup cloudnative-pg ( https://github.com/cloudnative-pg/cloudnative-pg ) yesterday.

I also recommend the Kubegres operator: https://www.kubegres.io/

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#68

Post by siamese_puff »

siliconc0w wrote: post #29

I wonder why more don't take advantage of native k8s and just rely on it to move over the persistent volume and start the new pod. This may have some small amount of downtime but it's a lot less complicated.

That gets complicated when you're running an HA cluster and need to worry about write conflicts.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#70

Post by mplewis »

gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

My advice is pay someone to run the cluster, and you can use all that time you saved on k8s complexity to operate the stuff inside the cluster and save some money.

Features like k8s Operators are essentially professionally-written code that stands in for a human agent that would take actions such as managing DB nodes, renewing certs, and performing backups. If you use mature operators, you can save a lot of money for a bit of effort.

Post reply on HN