Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Databases on Kubernetes

questdb.io

Post reply on HN
Print view View original

61–70 of 78 posts

Re: Running Databases on Kubernetes

#61

Post by twblalock »

debarshri wrote: post #32

I used to work for an org that deployed 3rd party legaltech "apps" on kubernetes which had all batteries included - Postgres, rabbitmq, redis, you name it I have seen it. Running statefulset even with the best operator there with a team of 4 is nothing short of a nightmare. Couple this with stability of rook ceph. In 2019, every operator had crazy bugs, we inherited all of them. You have to solve not just databases l…

One of the big problems with Kubernetes in general, especially back in 2019, is the alpha quality of almost everything in the ecosystem. Especially service meshes.

It seems to get worse the further down the stack you go. I’ve seen tons of problems with operators, monitoring tools, and CNIs.

It’s somewhat better now but there is still a lot of stuff you can’t depend on. The CNCF seems to endorse pretty much anything even if it’s crap.

Re: Running Databases on Kubernetes

#62

Post by worldsayshi »

tiimbz wrote: post #51
worldsayshi wrote: post #4

I've recently worked with putting postgres into kubernetes using the zalando operator. The impression has been such a mixed bag that it looks like we need to start over with some other operator. When we run into problems the documentation, error messages and configuration structure has been quite cryptic. Does anyone have any specific recommendations on what to use (like which operator) when setting up a postgres clu…

What type of issues did you run into? We've been running Zalando Postgres Operator for all our prod and dev clusters (around 100 in total) and couldn't be happier.

My impression is that when it works it works well but when it doesn't it doesn't help you that much. We have had two main issues:

- Can't set up two separate clusters in the same kubernetes instance because some cluster specific configuration is inexplicably set globally in the operator.

- Documentation and error messages are cryptic. Have to do a lot of trial and error to compensate for that. Maybe the issue here is a lack of experience with the stack used. Like Spilo and Patroni.

Re: Running Databases on Kubernetes

#63

Post by guipsp »

akdor1154 wrote: post #52
docandrew wrote: post #39

Earlier quoted context omitted.

This is really the secret - once someone figures out how to tie all the different k8s concepts into a functioning system, you can just copy it and put it on your cluster and it will probably work. Trying to figure it all out the first time is the messy part. If there’s an operator or Helm chart or something that does what you want there’s no shame in using it!

That's kind of the issue with the db question - for your app logic, 'will probably work' is fine. Playing fast and loose with your database is less enticing - you more likely want to understand every bit of the stack between you and your db, or if not, at least have a support line to whinge at if you hit trouble.

EDB provides commercial support for this

Re: Running Databases on Kubernetes

#64

Post by nly »

unity1001 wrote: post #7
advisedwang wrote: post #3

Not to diminish the product that QuestDB is working on, but another solution that works very well with Kubernetes is Vitess. Vitess is basically sharded MySQL, but it automatically manages this very well and has built in kubernetes support so it really handles the "pets to cattle" thing well.

> "pets to cattle" That phrase is inaccurate. With the cloud and K8, the pets move from being software that is tightly tied to the hardware to being a collection of configurations and software that are tightly tied to themselves. We just make the actual physical hardware anonymous. But from the perspective of the actual stack, there is still a server with its cpus, filesystem, i/o and everything. "Pets that you can c…

I really like this analogy

Re: Running Databases on Kubernetes

#65

Post by seanhunter »

yesbabyyes wrote: post #18
barrkel wrote: post #17

Earlier quoted context omitted.

What if you have finance customers who don't like commingled data, and you want to sell them a service and tell them with a straight face that their tenanted database isn't one bad query from serving up their data to someone else?

You can still have separate ACLs, databases, tables and even row level access control even if you share database servers.

Speaking as someone who has sold Saas into banks numerous times, "no shared tenancy" is very frequently an absolute and non-negotiable requirement.

Re: Running Databases on Kubernetes

#66

Post by yesbabyyes »

seanhunter wrote: post #65
yesbabyyes wrote: post #18

Earlier quoted context omitted.

You can still have separate ACLs, databases, tables and even row level access control even if you share database servers.

Speaking as someone who has sold Saas into banks numerous times, "no shared tenancy" is very frequently an absolute and non-negotiable requirement.

Fair enough; as 'twblalock says, it doesn't have to be more arduous to set up database servers per seat outside of k8s.

I simply wanted to highlight that there are many ways to skin a cat, and "no shared tenancy" is not in itself a valid argument for hosting your DBs in k8s, even though there may be other good reasons.

Re: Running Databases on Kubernetes

#67

Post by moondev »

_cenw wrote: post #54
marcinzm wrote: post #30

Earlier quoted context omitted.

Can’t you just use this then: https://aws.amazon.com/blogs/containers/aws-controllers-for-...

If you want to use k8s as your dataplane, sure. Though I'd rather entrust that task to Crossplane. There's always a point at which running things on Kubernetes becomes worth it. RDS is an expensive service, and if you want to enable every developer to boot up a development environment, or several even, if becomes prohibitivly expensive to use one process for everything. This is also true about running any workloads o…

It's the same RDS though. These controllers enable you to manage the lifecycle of the exact same AWS services you get when using the AWS console or CLI. Kubernetes is the porcelain and AWS offerings are still the plumbing.

Re: Running Databases on Kubernetes

#68

Post by smarterclayton »

GauntletWizard wrote: post #53
solatic wrote: post #44

Earlier quoted context omitted.

> k8s should run already with automatic node upgrades This is difficult to impossible to do with databases; even if your database has a built-in recovery method for when a primary is taken offline, in such a way that allows for zero-downtime in theory, the reality is that such mechanisms depend on the secondary staying online until the failover mechanism is complete. If you turn over control of node upgrades to the c…

Kubernetes does have this capability - Pod Disruption Budgets. They're underutilized and under tested, but at least the default cluster autoscaler respects them and will avoid destroying nodes that would break that constraint.

They are pretty well tested as of today (now that multiple vendors respect it during node upgrade), but now they’re relatively under featured for the next set of problems:

1. No way to signal that the workload is ready to accept traffic but not ready for disruption (such as an async background refill)

2. No way to provide backpressure over a time window, vs just instantaneous (ie during a rolling node upgrade PDB doesn’t prevent you from being moved multiple times)

3. No way for an admin and workload owner to coordinate on the amount of disruption a particular disruption will cause and discriminate between light and heavy (restart on node, recreate on node, move across node)

But yes, everyone with a nominally HA workload should have a PDB.

Re: Running Databases on Kubernetes

#69

Post by bwarminski »

matesz wrote: post #56
bwarminski wrote: post #55

This article does a great job describing the investment required to pull this off. At HubSpot, my team is running a large Vitess/MySql deployment (500+ distinct databases some sharded, multi region) atop k8s today and had to learn a lot of those same lessons and primitives. We opted to write our own operator(s) to do it. In the end, the investment has paid off in terms of being able to build self service functionalit…

The first question which comes to my mind is what are performance implications of running database like you do inside k8s vs EC2 vs bare metal? And how did you solve multitenancy? Does the operator handle lifecycle of database per customer simply or is it something more complicated? ps. And how you deal with migrations? ps. Forgive me if I'm asking for too much!

No worries, happy to share more details. For the databases where performance is a concern, we use constraints and reservation requests to all but guarantee it will be the only tenant on the node and we actively monitor CPU throttle and will autoscale in cases where it is sustained for a long period of time. We're actually achieving better overall utilization with this setup vs bare metal and arent dealing with a lot of issues with resource contention.

There is a main operator responsible for all the databases. It handles configuration changes, provisioning pods and slowly rolling out changes. In kube we model this with a custom resource we've defined called a KeyspaceShard which represents a named set of database instances that should participate in replication together. Once provisioned, the pods know how to hook up and detach from Vitess without requiring further involvement from the operator. Vitess handles backups and maintains the replication topology. "Complicated" is an apt description of what it does but not "complex". Evicting a database pod and letting the system reschedule and converge is a routine operation that doesn't cause much concern.

Migrations are done with gh-ost, which has its own custom operator that manages the lifecycle of the migration and ties into self service tooling we provide that is integrated with our build and deploy system.

Re: Running Databases on Kubernetes

#70

Post by potamic »

_skel wrote: post #10

I don't think the upsides are worth all the work. You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2? Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs.…

It's also more resource efficient, especially for non-production or non-critical workloads. VMs only come in discrete configurations and many times even the smallest one is too big, wasting a lot of resource. When you run thousands of instances, thanks to the magic of microservices, the costs add up.
Post reply on HN