31–40 of 78 posts
I've been quite happy with CloudNativePG on k8s. It was simple for me to set up on a k8s cluster with one primary and two replicas, if the primary box goes down another instance becomes primary, deal with connection pooling, and simple to have backups go to a cloud object store. The alternative is dealing with all the replication manually, making sure that your leader election and failover work, making sure you can s…
In 2019, every operator had crazy bugs, we inherited all of them. You have to solve not just databases level error but also errors popping from operators. If you can avoid databases on kubernetes, you should just do it.
Earlier quoted context omitted.
Re out of order: Is https://kubernetes.io/docs/concepts/workloads/controllers/st... unsuitable for that?
Unfortunately, while rolling updates account for some scenarios, they are not sufficient for handling out of order restarts where the order cannot be pre-determined. There’s probably some hack you could build with partitioning to mostly address the cases I am thinking of, but it isn’t elegant or guaranteed correct. This will be a problem for any database where clustering is synchronous and a specific primary node mus…
If a specific primary must start first, that’s partially what ordinals were intended to allow (0 is your primary, the others are always 1-N, and kube is responsible for ensuring the primary is never reassigned). I’d love to take feedback about places where the primitives are unusable, or ways they can be improved, because there are always new tools to add.
Earlier quoted context omitted.
Unfortunately, while rolling updates account for some scenarios, they are not sufficient for handling out of order restarts where the order cannot be pre-determined. There’s probably some hack you could build with partitioning to mostly address the cases I am thinking of, but it isn’t elegant or guaranteed correct. This will be a problem for any database where clustering is synchronous and a specific primary node mus…
Since I helped design them, I take some issue with that :). Certainly we never expected they would completely solve problems for the database, but they were definitely intended to provide guarantees that simplify normal consensus operations and prevent accidental confusion with non perfect databases. If a specific primary must start first, that’s partially what ordinals were intended to allow (0 is your primary, the…
Earlier quoted context omitted.
Re out of order: Is https://kubernetes.io/docs/concepts/workloads/controllers/st... unsuitable for that?
Unfortunately, while rolling updates account for some scenarios, they are not sufficient for handling out of order restarts where the order cannot be pre-determined. There’s probably some hack you could build with partitioning to mostly address the cases I am thinking of, but it isn’t elegant or guaranteed correct. This will be a problem for any database where clustering is synchronous and a specific primary node mus…
To handle this case, some teams may have a manual runbook, some teams may have some automation with ansible, some teams may have nothing. So, if someone can come up with some hacks and package that into a k8s operator, it is still a win. It seems to be the best primitives we have at the moment.
Earlier quoted context omitted.
Unfortunately, while rolling updates account for some scenarios, they are not sufficient for handling out of order restarts where the order cannot be pre-determined. There’s probably some hack you could build with partitioning to mostly address the cases I am thinking of, but it isn’t elegant or guaranteed correct. This will be a problem for any database where clustering is synchronous and a specific primary node mus…
You are talking about things like the "All Nodes Go Down Without a Proper Shutdown Procedure" example in https://galeracluster.com/library/documentation/crash-recove... right? To handle this case, some teams may have a manual runbook, some teams may have some automation with ansible, some teams may have nothing. So, if someone can come up with some hacks and package that into a k8s operator, it is still a win. It see…
The only thing I can think of is cost. My usage probably isn't high enough where there is any financial benefit to an alternative... but if it was, maybe I'd be considering this.
I have seen it fail way too many times. Inspecting a failing deployment that now has some magic Go code someone wrote running on this cluster. I can see using the basic kube building blocks: deployments, pods, config maps, etc.; there are enough guides and tools to help you out. As soon as you start writing code that runs in there, you're now dealing with two problems: your actual thing you're deploying, and now the operator.
Well, and then you need a mesh, and a way to manage certificates. and if it's a database to manage all the volumes. Everything looks good at the architect level - all the boxes and arrows line up, but when it breaks in production it's a nightmare to debug.
I've been quite happy with CloudNativePG on k8s. It was simple for me to set up on a k8s cluster with one primary and two replicas, if the primary box goes down another instance becomes primary, deal with connection pooling, and simple to have backups go to a cloud object store. The alternative is dealing with all the replication manually, making sure that your leader election and failover work, making sure you can s…
Earlier quoted context omitted.
> Use a helm chart and just bring your own little database for dev test and e2e tests. dev, test, and e2e tests should be done against full-size db clones
> dev, test, and e2e tests should be done against full-size db clones Real customer/sensitive data should not exist outside prod (and backup). So generally no, not full-size clones. I'd argue instrumentation in prod should give information on performance - for some tests/development you might need prod-size fake data.