Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Databases on Kubernetes

questdb.io

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Running Databases on Kubernetes

#21

Post by twblalock »

yesbabyyes wrote: post #18
barrkel wrote: post #17

Earlier quoted context omitted.

What if you have finance customers who don't like commingled data, and you want to sell them a service and tell them with a straight face that their tenanted database isn't one bad query from serving up their data to someone else?

You can still have separate ACLs, databases, tables and even row level access control even if you share database servers.

You can also have dedicated database servers per customer on bare metal or VMs. It's not always more work than asking your database team to learn Kubernetes...

Re: Running Databases on Kubernetes

#22

Post by samokhvalov »

axlee wrote: post #20
samokhvalov wrote: post #11

Earlier quoted context omitted.

> Use a helm chart and just bring your own little database for dev test and e2e tests. dev, test, and e2e tests should be done against full-size db clones

> dev, test, and e2e tests should be done against full-size db clones that's cute, what is your "full-size"? I don't have 2 days to run a test, and I'm pretty sure every single compliance requirements we are following would get obliterated the second someone hears about us doing that

Agreed on compliance part – of course, in many cases (not in all though) PII must not be in dev/test envs.

Although, cannot agree with the former part. If your tests are running 2 days on a full size clone, and it's an OLTP case, what about users, do they suffer from long query duration too? It sounds like it's time to optimize queries and/or redesign test sets (or both).

If it's bad in testing, it will be bad in prod. That's the idea of testing.

Example: how do you check schema changes?

Re: Running Databases on Kubernetes

#23

Post by samokhvalov »

Blackthorn wrote: post #19
samokhvalov wrote: post #11

Earlier quoted context omitted.

> Use a helm chart and just bring your own little database for dev test and e2e tests. dev, test, and e2e tests should be done against full-size db clones

You think I'm going to clone a multiple petabyte database just to run some tests?

not sure about petabytes (yet), but for dozens of TiB, we are fine – DB branching, thin clones, CoW are to the rescue.

Re: Running Databases on Kubernetes

#24

Post by annexrichmond »

_skel wrote: post #10

I don't think the upsides are worth all the work. You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2? Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs.…

yeah but if your org has orchestration tooling built around k8s, in a way it becomes much easier to provision a DB with k8s, setup the service, routing, networking, roles, etc than it would be in terraform. especially if you have to repeat this process in like 20 envs (stage, prod) x multiple regions

Re: Running Databases on Kubernetes

#25

Post by samokhvalov »

holografix wrote: post #13
e12e wrote: post #12

Earlier quoted context omitted.

> dev, test, and e2e tests should be done against full-size db clones Real customer/sensitive data should not exist outside prod (and backup). So generally no, not full-size clones. I'd argue instrumentation in prod should give information on performance - for some tests/development you might need prod-size fake data.

Couldn’t agree more. Having full sized and fully speced dev and test DBs is wasteful and not realistic across several independent teams. Monitoring prod closely and understanding what could constitute a costly workload/query which would in turn require a temporary test env with similar sized dataset is the correct approach.

This is too reactive approach, isn't it?

Imagine a system that could do this in CI/CD pipelines:

1) first, run all the tests on a tiny DB as usual 2) extract queries 3) run them against full-size DB branch/thin clone (thin provisioning, CoW; PII is not there of course, wiped out for security/compliance) -- auto-guessing parms, that's the trickiest part, but assume it's solved 4) collect all the details about performance, focusing on IO numbers (rows, block read/writes) 5) if some queries are off – say, you forgot LIMIT – post a warning, block the change, do not allow deploying it, letting backend dev fix it.

This would be proactive. And it's becoming possible with modern tools.

Re: Running Databases on Kubernetes

#26

Post by stonogo »

annexrichmond wrote: post #24
_skel wrote: post #10

I don't think the upsides are worth all the work. You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2? Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs.…

yeah but if your org has orchestration tooling built around k8s, in a way it becomes much easier to provision a DB with k8s, setup the service, routing, networking, roles, etc than it would be in terraform. especially if you have to repeat this process in like 20 envs (stage, prod) x multiple regions

This sounds dangerously close to "yeah but if the only tool you know is a hammer..."

Re: Running Databases on Kubernetes

#27

Post by tristor »

StatefulSet and PVCs aren’t sufficient to fully handle all the likely resilience challenges of running a database cluster on K8S. There needs to be some rethinking on how StatefulSet works to make it more appropriate to this use case, such as allowing Pods to be started out of order when recovering from failures.

I worked in this problem space extensively until 2020, and I think that there are paths forward but they require changes in K8S that none of the folks involved seem motivated to make. Realistically to make databases in K8S work well today you need a database built for K8S rather than one adapted for K8S.

The building blocks present today are not fundamentally capable of building a positive UX for adapting existing databases to K8S, but this is something that is worth making possible and I hope the community gets there some day.

Re: Running Databases on Kubernetes

#28

Post by smarterclayton »

tristor wrote: post #27

StatefulSet and PVCs aren’t sufficient to fully handle all the likely resilience challenges of running a database cluster on K8S. There needs to be some rethinking on how StatefulSet works to make it more appropriate to this use case, such as allowing Pods to be started out of order when recovering from failures. I worked in this problem space extensively until 2020, and I think that there are paths forward but they…

Re out of order:

Is https://kubernetes.io/docs/concepts/workloads/controllers/st... unsuitable for that?

Re: Running Databases on Kubernetes

#29

Post by tristor »

smarterclayton wrote: post #28
tristor wrote: post #27

StatefulSet and PVCs aren’t sufficient to fully handle all the likely resilience challenges of running a database cluster on K8S. There needs to be some rethinking on how StatefulSet works to make it more appropriate to this use case, such as allowing Pods to be started out of order when recovering from failures. I worked in this problem space extensively until 2020, and I think that there are paths forward but they…

Re out of order: Is https://kubernetes.io/docs/concepts/workloads/controllers/st... unsuitable for that?

Unfortunately, while rolling updates account for some scenarios, they are not sufficient for handling out of order restarts where the order cannot be pre-determined. There’s probably some hack you could build with partitioning to mostly address the cases I am thinking of, but it isn’t elegant or guaranteed correct.

This will be a problem for any database where clustering is synchronous and a specific primary node must start first on a full cluster restart. There are other out of band hacks you can do with reassigning PVCs, but it’s never elegant in the current primitives provided.

During my work in this problem space I became convinced that primitives for stateful applications in K8S were built specifically without considering databases as a valid use case. Everything else is just hacks after the fact to make it “work”.

Re: Running Databases on Kubernetes

#30

Post by marcinzm »

annexrichmond wrote: post #24
_skel wrote: post #10

I don't think the upsides are worth all the work. You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2? Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs.…

yeah but if your org has orchestration tooling built around k8s, in a way it becomes much easier to provision a DB with k8s, setup the service, routing, networking, roles, etc than it would be in terraform. especially if you have to repeat this process in like 20 envs (stage, prod) x multiple regions

Can’t you just use this then: https://aws.amazon.com/blogs/containers/aws-controllers-for-...
Post reply on HN