Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Databases on Kubernetes

questdb.io

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Running Databases on Kubernetes

#51

Post by tiimbz »

worldsayshi wrote: post #4

I've recently worked with putting postgres into kubernetes using the zalando operator. The impression has been such a mixed bag that it looks like we need to start over with some other operator. When we run into problems the documentation, error messages and configuration structure has been quite cryptic. Does anyone have any specific recommendations on what to use (like which operator) when setting up a postgres clu…

What type of issues did you run into?

We've been running Zalando Postgres Operator for all our prod and dev clusters (around 100 in total) and couldn't be happier.

Re: Running Databases on Kubernetes

#52

Post by akdor1154 »

docandrew wrote: post #39
mdasen wrote: post #6

I've been quite happy with CloudNativePG on k8s. It was simple for me to set up on a k8s cluster with one primary and two replicas, if the primary box goes down another instance becomes primary, deal with connection pooling, and simple to have backups go to a cloud object store. The alternative is dealing with all the replication manually, making sure that your leader election and failover work, making sure you can s…

This is really the secret - once someone figures out how to tie all the different k8s concepts into a functioning system, you can just copy it and put it on your cluster and it will probably work. Trying to figure it all out the first time is the messy part. If there’s an operator or Helm chart or something that does what you want there’s no shame in using it!

That's kind of the issue with the db question - for your app logic, 'will probably work' is fine. Playing fast and loose with your database is less enticing - you more likely want to understand every bit of the stack between you and your db, or if not, at least have a support line to whinge at if you hit trouble.

Re: Running Databases on Kubernetes

#53

Post by GauntletWizard »

solatic wrote: post #44
Felminor wrote: post #5

That's just a really really bad write-up on the real problem on running a database on k8s. You need ha because k8s should run already with automatic node upgrades. You need a pod disruption budget to make sure it is running and switching over when a node fails or gets upgraded. You want to either totally Oberprovision on memory or look into keep 2400 to make sure to fine-tune memory before k8s starts to throw your da…

> k8s should run already with automatic node upgrades This is difficult to impossible to do with databases; even if your database has a built-in recovery method for when a primary is taken offline, in such a way that allows for zero-downtime in theory, the reality is that such mechanisms depend on the secondary staying online until the failover mechanism is complete. If you turn over control of node upgrades to the c…

Kubernetes does have this capability - Pod Disruption Budgets. They're underutilized and under tested, but at least the default cluster autoscaler respects them and will avoid destroying nodes that would break that constraint.

Re: Running Databases on Kubernetes

#54

Post by _cenw »

marcinzm wrote: post #30
annexrichmond wrote: post #24

Earlier quoted context omitted.

yeah but if your org has orchestration tooling built around k8s, in a way it becomes much easier to provision a DB with k8s, setup the service, routing, networking, roles, etc than it would be in terraform. especially if you have to repeat this process in like 20 envs (stage, prod) x multiple regions

Can’t you just use this then: https://aws.amazon.com/blogs/containers/aws-controllers-for-...

If you want to use k8s as your dataplane, sure. Though I'd rather entrust that task to Crossplane.

There's always a point at which running things on Kubernetes becomes worth it. RDS is an expensive service, and if you want to enable every developer to boot up a development environment, or several even, if becomes prohibitivly expensive to use one process for everything. This is also true about running any workloads on Kubernetes to begin with though, and many companies definitely would be better off with a service like ECS/FarGate/CloudRun/Fly.io. Especially if they don't need the flexibility to build their own addon metrics/secrets/logging stacks.

Re: Running Databases on Kubernetes

#55

Post by bwarminski »

This article does a great job describing the investment required to pull this off. At HubSpot, my team is running a large Vitess/MySql deployment (500+ distinct databases some sharded, multi region) atop k8s today and had to learn a lot of those same lessons and primitives. We opted to write our own operator(s) to do it. In the end, the investment has paid off in terms of being able to build self service functionality for the rest of the business and write the kinds of tools workflows that allow us to support it with a relatively small team. The value is in the operator pattern itself and being able to manipulate things on a common control plane. Compared to the alternative of managing this with Terraform and Puppet/Ansible/Chef directly on EC2, which I've also done before, it's a better experience and much more maintainable even at the fixed expense of additional training and tooling.

I won't disagree with others that RDS is probably worth it until you need something very specific or have reached a certain scale.

Happy to share tips or pointers for anyone going down this path specifically with MySql or database workloads in general.

Re: Running Databases on Kubernetes

#56

Post by matesz »

bwarminski wrote: post #55

This article does a great job describing the investment required to pull this off. At HubSpot, my team is running a large Vitess/MySql deployment (500+ distinct databases some sharded, multi region) atop k8s today and had to learn a lot of those same lessons and primitives. We opted to write our own operator(s) to do it. In the end, the investment has paid off in terms of being able to build self service functionalit…

The first question which comes to my mind is what are performance implications of running database like you do inside k8s vs EC2 vs bare metal? And how did you solve multitenancy? Does the operator handle lifecycle of database per customer simply or is it something more complicated?

ps. And how you deal with migrations? ps. Forgive me if I'm asking for too much!

Re: Running Databases on Kubernetes

#57

Post by middle-marathon »

As a relative newcomer to k8s I was a bit surprised at the lack of backup tools available, coming from the world of on-prem Veeam which had more features than I knew what to do with. In my current role we had to find a way to back up our Postgres DBs running on k8s. We started using Kanister to actually take the backups but found there wasn't much around to actually manage the backups' lifecycle. I ended up writing Taweret (https://github.com/swissDataScienceCenter/taweret), a small tool which just ends up interacting with the Kanister CRDs to delete backups we no longer require based on a defined backups strategy.

Re: Running Databases on Kubernetes

#58

Post by ithkuil »

middle-marathon wrote: post #57

As a relative newcomer to k8s I was a bit surprised at the lack of backup tools available, coming from the world of on-prem Veeam which had more features than I knew what to do with. In my current role we had to find a way to back up our Postgres DBs running on k8s. We started using Kanister to actually take the backups but found there wasn't much around to actually manage the backups' lifecycle. I ended up writing T…

But that's what k8s is. It's not a tool that does a thing, but rather a set of APIs and patterns that let you glue together many tools that will let you a thing (for better or worse)

Re: Running Databases on Kubernetes

#59

Post by middle-marathon »

ithkuil wrote: post #58
middle-marathon wrote: post #57

As a relative newcomer to k8s I was a bit surprised at the lack of backup tools available, coming from the world of on-prem Veeam which had more features than I knew what to do with. In my current role we had to find a way to back up our Postgres DBs running on k8s. We started using Kanister to actually take the backups but found there wasn't much around to actually manage the backups' lifecycle. I ended up writing T…

But that's what k8s is. It's not a tool that does a thing, but rather a set of APIs and patterns that let you glue together many tools that will let you a thing (for better or worse)

Sure, I meant there wasn't really much around which ran on k8s to manage backups.

Re: Running Databases on Kubernetes

#60

Post by xyz-x »

We ran Zalando Operator for Postgres in k8s for a year, until finally succumbing to its technical debt that leaks out from every bit of its software being.

After switching to the Chrunchy Data pg operator v5 on k8s, we've had close to zero problems - one or two times a year the log shipping / HA replication fails and we have to restart it, but it's really neat! I can *warmly* recommend it; it really is CloudSQL in K8S.

Post reply on HN