Live data from Hacker News

PostgreSQL 14 on Kubernetes

blog.crunchydata.com

71–80 of 87 posts

Re: PostgreSQL 14 on Kubernetes

#71

Earlier quoted context omitted.

Because kubernetes is a fantastic platform that provides a lot of node and app management functionality out of the box. Stateful services like databases are not really a challenge anymore given recent developments in both K8S and database containerization.

With a database I generally want to have it on its own dedicated host with fast NVME drives, memory, cpu etc, maybe some OS specific system level configuration options for performance etc. I don't want to binpack it on a cluster of nodes each contending for system resources. Also RDBMS databases are not really designed to be ephemeral/elastic, they are stateful/persistent by nature. What does Kubernetes offer in this…

minio’s DirectCSI might be a solution you are interested in

Re: PostgreSQL 14 on Kubernetes

#72
post #3

Why would i want to run my database in k8s?

We run all our databases in K8S, approx 80 DBs. We're a fintech that requires HA so all databases run with at least 3 notes. Maging DB clusters with K8S only requires 1 file. If we need to increase storage this can be achieved in minutes, if we need to take one node down we basically can do it anytime during the day.. Every now and then we discuss moving DBs to AWS RDS (or equivalent) however for our transaction volu…

What DB are you running?

Re: PostgreSQL 14 on Kubernetes

#73
post #3

Why would i want to run my database in k8s?

We run all our databases in K8S, approx 80 DBs. We're a fintech that requires HA so all databases run with at least 3 notes. Maging DB clusters with K8S only requires 1 file. If we need to increase storage this can be achieved in minutes, if we need to take one node down we basically can do it anytime during the day.. Every now and then we discuss moving DBs to AWS RDS (or equivalent) however for our transaction volu…

I just checked your comment history and it appears that you like to troll.

How can I know that this isn't one of those cases? I know some folks in fintech and while obviously they're not _you_, they're definitely not the sort to be wishy washy with data, if anything they're quite conservative when it comes to how the databases are managed.

Any incidental complexity is incredibly discouraged.

Re: PostgreSQL 14 on Kubernetes

#74
post #69
post #62

Earlier quoted context omitted.

But that’s really not a great comparison, is it? Running on a VM is much more like running on a plain physical node. Running on K8s implies a whole new level of management overhead, that just isn’t there on a single node system. In this case, you should be asking “will we all be running databases in clustered mode in the future?” I think that answer still up for debate.

> Running on a VM is much more like running on a plain physical node. Sure, and containers are just plain processes on a VM. The exact code that controls processes in a VM, also controls processes inside a container. As long as the underlying data volume is reliable, it really doesn't matter whether you're running it outside of a container or not. Volumes for K8s have a come a long way since its early days.

But that’s not the difference. Any single process can run in a container without much issue. I think volume storage isn’t a solved issue yet, but it is better.

But running a database in K8s is implying (and it’s how the article sets it up) as a cluster. This is a big difference from running a solo instance on bare metal, in a VM, or in a container. That’s the point I was making to the parent comment.

To go up a level (comment), running Postgres in K8s doesn’t really get you much that you can’t also get from a VM (I’m thinking mainly about migration and setup). After that, the benefits come from running a cluster, which K8s can make easier to orchestrate over bare metal. But that’s a different architecture entirely.

Re: PostgreSQL 14 on Kubernetes

#76

Earlier quoted context omitted.

Say you have 100 database clusters and you want to schedule them on dedicated servers. A kubernetes node can advertise those local block devices as persistent volumes. Your StatefulSet can request that class of local storage and the kubernetes scheduler will then find a volume for the pods on a node that the pod can fit on. You then get all the features of kubernetes operators plus the performance of dedicated nodes…

> when a node disappears you get a new volume Actually no. Currently you need to manually delete the persistent volume claim. Followed by deleting the pod. For the statefulsets ti to create a new PVC. There even was a race condition until like 1.21 that made this operation fail 20% of the time so you'd have to delete the pod twice or the system would deadlock. It'd be nice if the PVC would claim a new PVC if the orch…

We do actually automate this away by deleting PVCs when they are tied to PVs on nodes with our drain taint on them and their pod is in the pending state. We also delete pods owned my StatefulSets that have missing PVCs. These two control loops really grease the wheels. It's definitely frustrating that statefulset pvc management isn't fully declarative, but hopefully this improves.

Re: PostgreSQL 14 on Kubernetes

#77

Earlier quoted context omitted.

> when a node disappears you get a new volume Actually no. Currently you need to manually delete the persistent volume claim. Followed by deleting the pod. For the statefulsets ti to create a new PVC. There even was a race condition until like 1.21 that made this operation fail 20% of the time so you'd have to delete the pod twice or the system would deadlock. It'd be nice if the PVC would claim a new PVC if the orch…

We do actually automate this away by deleting PVCs when they are tied to PVs on nodes with our drain taint on them and their pod is in the pending state. We also delete pods owned my StatefulSets that have missing PVCs. These two control loops really grease the wheels. It's definitely frustrating that statefulset pvc management isn't fully declarative, but hopefully this improves.

How do you prevent someone (or something) from accidentally draining a node hosting a database that does not have a synchronised replica?

Kubernetes is neat, but destroying stuff is so easy (and the context mechanism is just begging for human error) that I am a bit leery about hosting large amounts of stateful stuff in it.

Re: PostgreSQL 14 on Kubernetes

#78
post #66

Earlier quoted context omitted.

K8S runs workloads across servers in a declarative fashion. Ephemeral or stateful doesn't make a difference anymore. Whatever you use to manage the nodes directly is basically what K8S provides (and more) already, so you're really just replicating effort and complexity instead.

what happens when you declare that your PG 13 cluster should now be PG 14? I haven't yet seen good documentation on how these K8s databases handle upgrades. Additionally, they all seem very quick to stream from master to create a new node. That is great, for very, very small db's. But a 1TB db would be royal pain to do that with.

I think this is where the operator pattern really shines. By defining a custom resource that contains the cluster configuration, the operator can detect certain types of changes, such as upgrading Postgres to a new major version, and automate that change the same way that you'd do it manually. Of course, if such an operator doesn't already exist, its on you to build it, but with popular databases like Postgres, they are generally already out there in some form. That said, I'm not sure if existing Postgres operators handle your specific example.

Re: PostgreSQL 14 on Kubernetes

#79

Earlier quoted context omitted.

We do actually automate this away by deleting PVCs when they are tied to PVs on nodes with our drain taint on them and their pod is in the pending state. We also delete pods owned my StatefulSets that have missing PVCs. These two control loops really grease the wheels. It's definitely frustrating that statefulset pvc management isn't fully declarative, but hopefully this improves.

How do you prevent someone (or something) from accidentally draining a node hosting a database that does not have a synchronised replica? Kubernetes is neat, but destroying stuff is so easy (and the context mechanism is just begging for human error) that I am a bit leery about hosting large amounts of stateful stuff in it.

No database should lack an unsynchronized replica in general. A kubernetes admin could accidentally drain the host, but we run in AWS and AWS can kill nodes a lot faster than we can haha. We have enough servers in AWS that at least 5 fail per day. If some database is not replicated, someone is getting paged for it. We run MySQL with semi-sync replication for example: if a primary has two replicas, one replica must ack a write before the primary commits it, so you can lose the primary at any time and not lose data. That said, kubernetes has prevention mechanisms for this.

PodDisruptionBudgets allow you to specify the minimum number of replicas. The drain command calls evict instead of delete and evict honors these restrictions. This is helpful for node draining, but not necessarily helpful if someone accidentally deletes a StatefulSet. However, nothing in kubernetes deletes PVC objects automatically, so if you delete the pod, the volume remains bound to the PVC until you delete it.

preStop hooks are used in all the database pods to ensure that if the current pod is the primary, leadership is correctly changed to a replica before proceeding with the delete. We combine this with extremely long termination grace periods and alerting if termination is taking too long so that an ops team can look into the issue.

preStop hooks are great, but they also are somewhat of a time bomb if not careful, so some of our database operators make use of finalizers instead. With finalizers, nothing happens when a pod enters the terminating state until an operator marks the pod's finalizer as completed. This flow is really nice because you end up with behavior where the kubernetes cluster admins or built in controllers effectively are really gently requesting that a pod terminate, and it's up to the operator of that pod to decide exactly how and when that occurs.

For non-local volumes, like EBS PVCs, once the PVC is deleted an operator will snapshot the volume during a finalizer before allowing the volume to be fully deleted so that we can recover data if this ever accidentally occurs. This was the first protection we ever implemented when initially adopting kubernetes back when we had no idea what we were doing so that we could always recover data in the event of admin mistakes. We can't do this for local volumes, however, any team making use of local volumes truly needs to stay on top of replication when running in the cloud, whether on kubernetes or not or else you absolutely will lose data. AWS local nvme volumes have limited write cycles just like the ones in on-prem servers. So ideally all databses are configured to not accept writes that cannot be replicated. And even with EBS, at scale we end up with something like 20 unrecoverable volume failures per year

Re: PostgreSQL 14 on Kubernetes

#80

Earlier quoted context omitted.

> when a node disappears you get a new volume Actually no. Currently you need to manually delete the persistent volume claim. Followed by deleting the pod. For the statefulsets ti to create a new PVC. There even was a race condition until like 1.21 that made this operation fail 20% of the time so you'd have to delete the pod twice or the system would deadlock. It'd be nice if the PVC would claim a new PVC if the orch…

We do actually automate this away by deleting PVCs when they are tied to PVs on nodes with our drain taint on them and their pod is in the pending state. We also delete pods owned my StatefulSets that have missing PVCs. These two control loops really grease the wheels. It's definitely frustrating that statefulset pvc management isn't fully declarative, but hopefully this improves.

Is this part of the postgres controller or is this an own controller that you wrote?
Post reply on HN