Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL 14 on Kubernetes

blog.crunchydata.com

Post reply on HN
Print view View original

81–87 of 87 posts

Re: PostgreSQL 14 on Kubernetes

#81

Post by chousuke »

paulfurtado wrote: post #79
chousuke wrote: post #77

Earlier quoted context omitted.

How do you prevent someone (or something) from accidentally draining a node hosting a database that does not have a synchronised replica? Kubernetes is neat, but destroying stuff is so easy (and the context mechanism is just begging for human error) that I am a bit leery about hosting large amounts of stateful stuff in it.

No database should lack an unsynchronized replica in general. A kubernetes admin could accidentally drain the host, but we run in AWS and AWS can kill nodes a lot faster than we can haha. We have enough servers in AWS that at least 5 fail per day. If some database is not replicated, someone is getting paged for it. We run MySQL with semi-sync replication for example: if a primary has two replicas, one replica must ac…

Thanks for the detailed response. I was aware Kubernetes has hooks for pretty much everything, so getting details on how they are used in practice is interesting.

Mostly I'm concerned about preventing administrator mistakes; if you have enough privileges, it's distressingly easy to just accidentally delete all kinds of resources in the default configuration, especially since kubectl is context-sensitive. I like my automation software to have checks in it to prevent me from doing stupid things without explicitly disabling a number of safeties first.

I feel like Kubernetes is a powertool that requires a competent and trained admin team or managed access so that you simply can't make awful mistakes as a "regular user", but a lot of the hype around it seems to be focused on how "easy" it is.

To my eyes, it's indeed easy to just throw manifests from the internet at Kubernetes, but that is often akin to disabling SELinux because some blog says so; it may get things working, but it's not the competent choice most of the time.

I'm also a big proponent of storing everything in git repositories, so I do like how Kubernetes enables declarative configuration, though it seems much of the tooling in the ecosystem works against that by just creating resources in a cluster with no version control...

Re: PostgreSQL 14 on Kubernetes

#82

Post by paulfurtado »

chousuke wrote: post #81
paulfurtado wrote: post #79

Earlier quoted context omitted.

No database should lack an unsynchronized replica in general. A kubernetes admin could accidentally drain the host, but we run in AWS and AWS can kill nodes a lot faster than we can haha. We have enough servers in AWS that at least 5 fail per day. If some database is not replicated, someone is getting paged for it. We run MySQL with semi-sync replication for example: if a primary has two replicas, one replica must ac…

Thanks for the detailed response. I was aware Kubernetes has hooks for pretty much everything, so getting details on how they are used in practice is interesting. Mostly I'm concerned about preventing administrator mistakes; if you have enough privileges, it's distressingly easy to just accidentally delete all kinds of resources in the default configuration, especially since kubectl is context-sensitive. I like my au…

Cool thing about finalizers is that there's no straightforward way to delete them from the CLI, you need to explicitly patch the object to delete the finalizer so that goes a long way towards preventing instant mass deletion, they really do help a lot here.

The additional protection we have is via admission controllers which hook all API calls to prevent mistakes, of course, you have to foresee those mistakes and come up with a validation that prevents it. But even something as simple as "only 5 pods are allowed to be terminating at once" can help.

I absolutely agree about people claiming kubernetes is easy. It is easy to throw some helm chart at the kube API and suddenly have an app running with all it's databses configured. It is hard to keep them all running.

The pattern of operators seeks to solve this. Instead of a helm chart creating a MySQL statefulset, it creates a MySQLCluster that gets managed by a well-written operator following the best practices and correctly implementing all the hooks. The ecosystem is starting to converge on this finally and the beauty is that the sum of the industry's operational knowledge can be coded into these operators and we'll finally arrive at nearly fully self managing databases with backups and all.

The industry isn't there quite yet though, and so at least for stateful workloads you need significant operational experience. For background: I am the Tech Lead of our team that runs the kubernetes clusters and we provide all of the primitives our users need and control access. Each database type then has dedicated teams operating them and codifying operations into operators. There is a lot of work being done. But prior to kubernetes, all of these teams separately were coding against the EC2 API with no standardization, no unified view, ad-hoc failure detection, ad-hoc auditing, etc. Kubernetes is a very substantial improvement over that, but 90% of companies never reach a scale where this is necessary. But once extremely solid open source operators exist we may truly hit the dream of just applying some self managing operator manifest to an EKS or GKE cluster and getting an actually production grade database setup.

Version control is an interesting topic because it's hard to express transitive dependencies in that code. For example, the MySQL operator creating a StatefulSet and the Statefulset creating pods. It doesn't make sense to commit those lower level resources to git, they're not fully independent. However, for top level things, our build system produces artifacts from git and the deploy system creates them in the cluster. With this setup, only admins could directly apply them, which really helps with keeping things in sync with git.

Re: PostgreSQL 14 on Kubernetes

#83

Post by morelisp »

cdogl wrote: post #48
morelisp wrote: post #26

Earlier quoted context omitted.

If you choose a) you don't really get to avoid b), since something still has to bootstrap your Kubernetes nodes themselves. So it's more a question whether you'd rather write a Kubernetes manifest or a Salt/Ansible config; and with the complexity and limitations of a lot of storage operators, and no real "orchestration" benefit once you have pinned deployments to particular nodes anyway, sometimes it's the latter. Th…

> If you choose a) you don't really get to avoid b), since something still has to bootstrap your Kubernetes nodes themselves Not quite true if you use a managed K8S offering - and it seems a self-managed cluster is something to be avoided until absolutely necessary. Even with a managed cluster like GKE there's plenty of operational overhead to keep everything up-to-date and working properly.

If you're running on a managed K8s offering yet not using the managed storage, you're getting ripped off. (I'm well aware this describes a lot of companies / projects.)

Re: PostgreSQL 14 on Kubernetes

#84

Post by morelisp »

hijinks wrote: post #57
leetrout wrote: post #3

Why would i want to run my database in k8s?

20 years ago people said the same thing about running DBs on virtual machines. Now almost all databases are run on VMs

> Now almost all databases are run on VMs

And now almost all databases (meaning running instances, not software artifacts) are way slower than necessary and lie about durability.

Re: PostgreSQL 14 on Kubernetes

#85

Post by morelisp »

jatone wrote: post #64
mbreese wrote: post #62

Earlier quoted context omitted.

But that’s really not a great comparison, is it? Running on a VM is much more like running on a plain physical node. Running on K8s implies a whole new level of management overhead, that just isn’t there on a single node system. In this case, you should be asking “will we all be running databases in clustered mode in the future?” I think that answer still up for debate.

containers are actually a really good fit for dbs. they do one thing and one thing only and for a really really long time without many changes. and the answer is we will definitely be running DBs in clustered mode in the future. its not really a debate its an eventuality. without clusters you lose on reliability and maintenance QOL improvements. without clusters its very hard to upgrade systems effectively leaving sy…

The question isn't whether DBs will be running in clusters - they all pretty much do, or can - but will those clusters be orchestrated by something first designed for dynamic scaling of stateless services, treating all nodes as varying-sized pools of otherwise mostly-homogeneous hyperconverged compute/storage? Or, how much are you willing to pay in $/performance/dev time to force them into such infrastructure?

Or will they be on something more purpose-built, either designed from the start to handle storage more flexibly at a higher level? Or conversely something more specific and lower-level to extract maximum performance for their particular storage patterns?

Re: PostgreSQL 14 on Kubernetes

#86

Post by paulfurtado »

arianvanp wrote: post #80
paulfurtado wrote: post #76

Earlier quoted context omitted.

We do actually automate this away by deleting PVCs when they are tied to PVs on nodes with our drain taint on them and their pod is in the pending state. We also delete pods owned my StatefulSets that have missing PVCs. These two control loops really grease the wheels. It's definitely frustrating that statefulset pvc management isn't fully declarative, but hopefully this improves.

Is this part of the postgres controller or is this an own controller that you wrote?

This is a controller we wrote in order to make local storage actually usable. Internally we have a "kube cluster operator" that handles the cluster-level ops like this, gracefully draining nodes, etc.

Re: PostgreSQL 14 on Kubernetes

#87

Post by chousuke »

paulfurtado wrote: post #82
chousuke wrote: post #81

Earlier quoted context omitted.

Thanks for the detailed response. I was aware Kubernetes has hooks for pretty much everything, so getting details on how they are used in practice is interesting. Mostly I'm concerned about preventing administrator mistakes; if you have enough privileges, it's distressingly easy to just accidentally delete all kinds of resources in the default configuration, especially since kubectl is context-sensitive. I like my au…

Cool thing about finalizers is that there's no straightforward way to delete them from the CLI, you need to explicitly patch the object to delete the finalizer so that goes a long way towards preventing instant mass deletion, they really do help a lot here. The additional protection we have is via admission controllers which hook all API calls to prevent mistakes, of course, you have to foresee those mistakes and com…

I'm a bit late replying, but I just want to clarify my stance on the last paragraph; I don't think you need to store everything in the cluster in version control if they're created automatically as a result of applying the top-level configuration, If there's something important that's too dynamic to be in source control, I treat it as data to be backed up.

My ideal is that what is in the repository allows me to rebuild the system from scratch (assuming suitable hardware exists) to the point where it can be successfully used as a restore target for your backups following documented restore procedures.

Post reply on HN