Live data from Hacker News

Running Databases on Kubernetes

questdb.io

71–78 of 78 posts

Re: Running Databases on Kubernetes

#71
post #27

StatefulSet and PVCs aren’t sufficient to fully handle all the likely resilience challenges of running a database cluster on K8S. There needs to be some rethinking on how StatefulSet works to make it more appropriate to this use case, such as allowing Pods to be started out of order when recovering from failures. I worked in this problem space extensively until 2020, and I think that there are paths forward but they…

>"I worked in this problem space extensively until 2020, and I think that there are paths forward but they require changes in K8S that none of the folks involved seem motivated to make."

Can you say what you see as those possible paths forward and what changes they would require?

Re: Running Databases on Kubernetes

#72
post #36

Earlier quoted context omitted.

You are talking about things like the "All Nodes Go Down Without a Proper Shutdown Procedure" example in https://galeracluster.com/library/documentation/crash-recove... right? To handle this case, some teams may have a manual runbook, some teams may have some automation with ansible, some teams may have nothing. So, if someone can come up with some hacks and package that into a k8s operator, it is still a win. It see…

That is exactly one of the scenarios I am thinking about. Yes, there are hacks with PVCs that make recovering from this possible today with StatefulSet.

In that scenario it looks like members must coordinate to identify the highest committed transaction (identifying the list of valid members) and then bootstrap from that member?

Stateful Sets were designed to standardize two hard problems: being able to identify all the valid members (pods identified by number that are running at most once on any node) and give admins the button to decide a member was never coming back (the force delete pod / force delete pv action). That was true black magic before - everyone did it their own way. So we worked with the ecosystem to enable vendors/communities/individuals to map those primitives into specific solutions, but did it somewhat deliberately as “we have to build this together”.

What I think the gap has been is that there is significant friction in between the three realms of expertise - knowing what kube is providing, knowing how to map that to a specific problem like translating the Galera runbook into operator/script logic, and then communicating that to the teams that will be accountable for reacting. Vendors have incentives to make you pay for that expertise (or may not have it), large organizations hire people to provide it (most large db on kube deployments are also large tech companies), and in between you have a lot of uncertainty and knowledge gaps that doesn’t necessarily transfer, and that is what drives “Kube isn’t great for stateful”.

It’s ironic to me because StatefulSets were intended to take advantage of those incentives to help the ecosystem scale, and the result is “worse is better” in that many more people can do state on HA DBs than were successful on VMs or metal, but it doesn’t mean they’re completely successful and when people hit the rough edges it hurts. We can do better (that’s partially my day job), but there is a lot of pain people have taken so far that was probably unnecessary. You should use managed DBs if you can - and when you don’t, Kube should be the best alternative it can be (which isn’t far - many DB SaaS uses some kube), and that’s what we need to focus on.

Re: Running Databases on Kubernetes

#73
post #34

Earlier quoted context omitted.

Since I helped design them, I take some issue with that :). Certainly we never expected they would completely solve problems for the database, but they were definitely intended to provide guarantees that simplify normal consensus operations and prevent accidental confusion with non perfect databases. If a specific primary must start first, that’s partially what ordinals were intended to allow (0 is your primary, the…

I am not working in this problem space any longer, but it’s likely we’ve crossed paths. I have previously presented some suggestions to the Storage SIG on this topic. Feel free to reach out to me with the info in my profile and I can get back to you with a more detailed write up on the specific challenges that I would like to see addressed in StatefulSet, however I am currently traveling so my response will be delaye…

>"I have previously presented some suggestions to the Storage SIG on this topic."

Ia there any chance you might be able to provide a link to these? I would be curious to take a look.

Re: Running Databases on Kubernetes

#74
post #56

Earlier quoted context omitted.

The first question which comes to my mind is what are performance implications of running database like you do inside k8s vs EC2 vs bare metal? And how did you solve multitenancy? Does the operator handle lifecycle of database per customer simply or is it something more complicated? ps. And how you deal with migrations? ps. Forgive me if I'm asking for too much!

No worries, happy to share more details. For the databases where performance is a concern, we use constraints and reservation requests to all but guarantee it will be the only tenant on the node and we actively monitor CPU throttle and will autoscale in cases where it is sustained for a long period of time. We're actually achieving better overall utilization with this setup vs bare metal and arent dealing with a lot…

Thanks for answering!

Re: Running Databases on Kubernetes

#75
post #44

Earlier quoted context omitted.

> k8s should run already with automatic node upgrades This is difficult to impossible to do with databases; even if your database has a built-in recovery method for when a primary is taken offline, in such a way that allows for zero-downtime in theory, the reality is that such mechanisms depend on the secondary staying online until the failover mechanism is complete. If you turn over control of node upgrades to the c…

Online lossless zero downtime upgrades? I've done it with Cassandra...and yeah Kafka can do it I've heard. But those can be 30 hour operations even with you ducks in a row, and you better have backup strategies ready. Fun story, Amazon said rds would be always be zero downtime upgrades. But then came a major version upgrade and .... Surprise it wasn't.

Add a new RDS replica, wait for it to sync, promote it to master? Zero downtime achieved

Re: Running Databases on Kubernetes

#76
post #70
post #10

I don't think the upsides are worth all the work. You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2? Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs.…

It's also more resource efficient, especially for non-production or non-critical workloads. VMs only come in discrete configurations and many times even the smallest one is too big, wasting a lot of resource. When you run thousands of instances, thanks to the magic of microservices, the costs add up.

Those are good arguments for ephemeral workloads but they don't make as much sense for databases.

Re: Running Databases on Kubernetes

#77

Earlier quoted context omitted.

Online lossless zero downtime upgrades? I've done it with Cassandra...and yeah Kafka can do it I've heard. But those can be 30 hour operations even with you ducks in a row, and you better have backup strategies ready. Fun story, Amazon said rds would be always be zero downtime upgrades. But then came a major version upgrade and .... Surprise it wasn't.

Add a new RDS replica, wait for it to sync, promote it to master? Zero downtime achieved

I didn't do the upgrade, I'm not a postgres MySQL person, but the best they could do was a third party tool that dropped it to a couple minutes.

Re: Running Databases on Kubernetes

#78
post #56

Earlier quoted context omitted.

The first question which comes to my mind is what are performance implications of running database like you do inside k8s vs EC2 vs bare metal? And how did you solve multitenancy? Does the operator handle lifecycle of database per customer simply or is it something more complicated? ps. And how you deal with migrations? ps. Forgive me if I'm asking for too much!

No worries, happy to share more details. For the databases where performance is a concern, we use constraints and reservation requests to all but guarantee it will be the only tenant on the node and we actively monitor CPU throttle and will autoscale in cases where it is sustained for a long period of time. We're actually achieving better overall utilization with this setup vs bare metal and arent dealing with a lot…

One more question - did you have a chance to see where scylladb is going?

On latest conference their CEO said it's all about serverless and virtualization. Having kubernetes doing all the work [1] - "we are doing it automatically for you with our management which is based on using multi-tenant kuebrentes deployment". Even more surprising is that instead of using NVMEs they want to use s3 for backend storage and NVME is only cache [2] :o I am not database expert, but this is very interesting.

[1] https://youtu.be/ZX7rA78BYS0?t=1303

[2] https://youtu.be/ZX7rA78BYS0?t=2086

Post reply on HN