Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-hosting a high-availability Postgres cluster on Kubernetes

ryan-schachte.com

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#71

Post by dilyevsky »

snapetom wrote: post #50
steveBK123 wrote: post #47

Earlier quoted context omitted.

Wouldn't most data driven, disk-IO intensive workloads be a bad fit for k8s? Especially when its not purely read-only.. That seems the crux of it.

Absolutely. I haven't gotten in the weeds of K8 volumes, but certainly have done a lot with Docker volumes including adding some PRs to fix some issues I've ran into. There is a lot under the hood with Docker volumes. With how DBs like to optimize their IO operations, I'm frankly a little amazed how it somehow mostly works between the DB -> orchestrator volumes -> file system.

It works totally fine on k8s with iscsi/fcoe/nvme-backed volumes provided remote targets themselves offer sufficient performance for your db workload

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#72

Post by dilyevsky »

levkk wrote: post #17
gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

Volume resizing support has been added since ~1.11 which is like 5 years old at this point

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#73

Post by dilyevsky »

oxfordmale wrote: post #43

No, just no. K8s shouldn't be used to host database systems. Its main function is micro services. Cloud provider provide managed versions of Postgres that are highly available. Even if you self host, Kubernetes isn't the answer.

Perfectly fit for running databases. HA databases that have built in replication (cockroach, tidb, foundationdb, etc) are easier, pg is just not built for proper clustering. Still doable tho

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#74

Post by dilyevsky »

doublerabbit wrote: post #39
bo0tzz wrote: post #25

Earlier quoted context omitted.

Wherever you want, Kubernetes supports basically every storage backend you can imagine.

> Kubernetes supports basically every storage backend you can imagine. Text files?

If you can figure out how to create mount point out of it and provide your own csi driver then yes

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#75

Post by p_l »

SlightlyLeftPad wrote: post #62
debarshri wrote: post #45

Earlier quoted context omitted.

Problem is that RDS comes at a price. It is purely about operation cost. When you have 1500+ databases these cost add up. At that point, this kind of techniques are required to self host the databases. Price per DB with HPA na VPA is way lower than what you would pay for managed databases as well as you can hire a full time devops+dbadmin and still be cheaper.

This is a perfect example of a cost savings opportunity that offers no benefit and is really just a foot gun when it comes to operational complexity. The cloud cost savings almost never justify the operational costs here. There’s typically a very long list of other cost optimizations to be made before this would ever be on the table for me. If you’re talking about 1500+ databases and think a single full time devops+d…

Except the typical cloud prices are dialed for USA, and it's not that unusual in my experience for the cloud savings to be high enough to add more than one highly skilled engineer who will give large positive impact all around.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#76

Post by p_l »

cwiggs wrote: post #54
steveBK123 wrote: post #47

Earlier quoted context omitted.

Wouldn't most data driven, disk-IO intensive workloads be a bad fit for k8s? Especially when its not purely read-only.. That seems the crux of it.

This seems to be the common advise given, but I don't fully agree. There have been many times in my career where a DB was on a VM with the storage attached via a cloud providers block storage. When asked if we should move it to k8s, people are quick to mention k8s doesn't do well with persistent storage. However all of the big cloud providers offer the ability to easily create a persistent volume in k8s that then jus…

Given k8s origin as bare-metal oriented system, attaching physical SAN volumes was there pretty early on, and now it's only more capable in that area (including passing devices for running exotic stuff)

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#77

Post by p_l »

snapetom wrote: post #50
steveBK123 wrote: post #47

Earlier quoted context omitted.

Wouldn't most data driven, disk-IO intensive workloads be a bad fit for k8s? Especially when its not purely read-only.. That seems the crux of it.

Absolutely. I haven't gotten in the weeds of K8 volumes, but certainly have done a lot with Docker volumes including adding some PRs to fix some issues I've ran into. There is a lot under the hood with Docker volumes. With how DBs like to optimize their IO operations, I'm frankly a little amazed how it somehow mostly works between the DB -> orchestrator volumes -> file system.

K8s mostly drops docker volumes code (and many other parts of docker, TBH). You can pass a random mount point or even a raw device and it will have only the overhead of kernel IO code for namespaces, overhead you hit anyway.

The only "docker graph" stuff is in OCI container data itself.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#78

Post by p_l »

gbartolini wrote: post #63
levkk wrote: post #17

Earlier quoted context omitted.

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

I don’t agree here. There are operators like the one I’m a maintainer of (CloudnativePG) which works directly with Kubernetes, teaching it how to handle Postgres clusters as a coordinated set of instances. Enormous improvements have been done in the last couple of years, and we are particularly focused in working together with storage groups in Kubernetes to handle database workloads, such as for example declarative…

Thanks for working on CNPG!

I have been a very happy user of CNPG even with occasional issues (database backup to GCS tripped me few times, but it works - mostly a bit of UX that I never was sure wasn't some fail of mine).

Now I only really need to add some automation for handling "recover the database and switch over clients to it" that is more automated (I understand why CNPG doesn't do recovery to existing database, but it is a bit annoying)

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#79

Post by p_l »

seanhunter wrote: post #46
marcosdumay wrote: post #36

Earlier quoted context omitted.

> What are the alternatives? Installing the Postgres on the computer, or on a VM. Really, just the fact that some people keep asking this question is enough to question everything else they say. The alternatives are obvious .

Exactly. The way people successfully did HA and scalability on databases for more than 20 years before Kubernetes existed.

With a bunch of hand crafted, hairy code to link the necessary features like fail over, backups, PITR, clustering.

I know, I did all of that.

Which is why I punt that effort to a k8s operator these days, because at the end of the day it does everything I did manually plus makes it easier for me to spin a new database that has WAL shipping and backups to different location.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#80

Post by p_l »

ssijak wrote: post #10

Is Kubernetes still hard in 2024?

Depends (I know, I know...)

If you learn it from the top, immediately jumping through bunch of deployments, Helm (ugh) and other stuff?

You'll get fast to deployment but you won't know how to deal with things failing and there will be a lot of stuff that will remain "magic". Eady way to end up cargo culting despite best intentions.

Can be enough of you're just making simple apps to run on it but not running the clusteror the application in production. But you will have hard time understanding why things work and everything will be complex upon complex.

Go from the bottom up, learn basics of kube API patterns, kubelet, how scheduling works (don't have to be in depth), how kubelet works, how networking works (CNI, why kube-proxy is a bandaid for applications that handle networking badly, how services work), how storage works (how kubelet mounts things to containers, etc). Then how higher level controllers (aka operators) work - from Pods, through ReplicaSet to Deployment, StatefulSet, DaemonSet.

This way you'll learn the basic building blocks, which are quite simple despite the long list I just gave, because the architectural and API patterns repeat and build over each other. The core is simple which lets you build complex stuff on top while still understanding it.

Post reply on HN