Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Databases on Kubernetes

questdb.io

Post reply on HN
Print view View original

11–20 of 78 posts

Re: Running Databases on Kubernetes

#11

Post by samokhvalov »

Felminor wrote: post #5

That's just a really really bad write-up on the real problem on running a database on k8s. You need ha because k8s should run already with automatic node upgrades. You need a pod disruption budget to make sure it is running and switching over when a node fails or gets upgraded. You want to either totally Oberprovision on memory or look into keep 2400 to make sure to fine-tune memory before k8s starts to throw your da…

> Use a helm chart and just bring your own little database for dev test and e2e tests.

dev, test, and e2e tests should be done against full-size db clones

Re: Running Databases on Kubernetes

#12

Post by e12e »

samokhvalov wrote: post #11
Felminor wrote: post #5

That's just a really really bad write-up on the real problem on running a database on k8s. You need ha because k8s should run already with automatic node upgrades. You need a pod disruption budget to make sure it is running and switching over when a node fails or gets upgraded. You want to either totally Oberprovision on memory or look into keep 2400 to make sure to fine-tune memory before k8s starts to throw your da…

> Use a helm chart and just bring your own little database for dev test and e2e tests. dev, test, and e2e tests should be done against full-size db clones

> dev, test, and e2e tests should be done against full-size db clones

Real customer/sensitive data should not exist outside prod (and backup). So generally no, not full-size clones. I'd argue instrumentation in prod should give information on performance - for some tests/development you might need prod-size fake data.

Re: Running Databases on Kubernetes

#13

Post by holografix »

e12e wrote: post #12
samokhvalov wrote: post #11

Earlier quoted context omitted.

> Use a helm chart and just bring your own little database for dev test and e2e tests. dev, test, and e2e tests should be done against full-size db clones

> dev, test, and e2e tests should be done against full-size db clones Real customer/sensitive data should not exist outside prod (and backup). So generally no, not full-size clones. I'd argue instrumentation in prod should give information on performance - for some tests/development you might need prod-size fake data.

Couldn’t agree more. Having full sized and fully speced dev and test DBs is wasteful and not realistic across several independent teams.

Monitoring prod closely and understanding what could constitute a costly workload/query which would in turn require a temporary test env with similar sized dataset is the correct approach.

Re: Running Databases on Kubernetes

#14

Post by tommyzli »

worldsayshi wrote: post #4

I've recently worked with putting postgres into kubernetes using the zalando operator. The impression has been such a mixed bag that it looks like we need to start over with some other operator. When we run into problems the documentation, error messages and configuration structure has been quite cryptic. Does anyone have any specific recommendations on what to use (like which operator) when setting up a postgres clu…

the last time I gave the Postgres operator space a serious look was about a year ago, and at the time the Zalando operator was far and away the most feature complete and mature.

We had a couple unusual requirements that the operator wasn't really suited for, so we ultimately ended up writing our own helm chart and forgoing the operator route altogether

Re: Running Databases on Kubernetes

#15

Post by ahachete »

The key for me is the level of automation that you can reach at a reasonable "development cost". Let me elaborate.

K8s, if anything, is an API. An API that allows you to interact with compute, storage and networks in a way that is abstracted from the actual underlying infrastructure. This is incredibly powerful. You can, essentially, code and automate all your infrastructure.

But this goes beyond deployment, something you could achieve (more or less) with tools like Terraform or Pulumi. Enter "Day 2 operations".

Day 2 operations are essential for any database. And cloud services have done a good job at automating them. Speaking of Postgres, my daily job, things like HA, backups but also minor and major version upgrades are table stakes day 2 operations.

If you want to build these day 2 operations in the cloud (say on VMs), even though you have APIs do to so, a) they don't implement a pattern like Kubernete's reconciliation cycle; and b) you have a distinct API per cloud. K8s solves both problems, making it way "cheaper" to build such an automation. On K8s, a given operator can code these day 2 operations against K8s APIs. Therefore, if you want to build such automation, either you are a cloud provider (and potentially do this only for your own cloud) or you do it on Kubernetes.

This is so much true, that existing operators have already gone beyond what DBaaS do. Speaking of StackGres [0] (disclaimer: founder), we have implemented day 2 operations (other than the "table stakes" ones that I mentioned before) that no other DBaaS offers as of today, such as vacuums, repacks and even benchmarks (and more day 2 operations will be developed). See [1] for the CRD specs of SGDbOps, our "Day 2 operations" if you are interested.

[0] https://stackgres.io [1] https://stackgres.io/doc/latest/reference/crd/sgdbops/

Re: Running Databases on Kubernetes

#16

Post by superyesh »

_skel wrote: post #10

I don't think the upsides are worth all the work. You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2? Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs.…

+1 Sometimes just because you can does not mean you should.

Re: Running Databases on Kubernetes

#17

Post by barrkel »

_skel wrote: post #10

I don't think the upsides are worth all the work. You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2? Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs.…

What if you have finance customers who don't like commingled data, and you want to sell them a service and tell them with a straight face that their tenanted database isn't one bad query from serving up their data to someone else?

Re: Running Databases on Kubernetes

#18

Post by yesbabyyes »

barrkel wrote: post #17
_skel wrote: post #10

I don't think the upsides are worth all the work. You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2? Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs.…

What if you have finance customers who don't like commingled data, and you want to sell them a service and tell them with a straight face that their tenanted database isn't one bad query from serving up their data to someone else?

You can still have separate ACLs, databases, tables and even row level access control even if you share database servers.

Re: Running Databases on Kubernetes

#19

Post by Blackthorn »

samokhvalov wrote: post #11
Felminor wrote: post #5

That's just a really really bad write-up on the real problem on running a database on k8s. You need ha because k8s should run already with automatic node upgrades. You need a pod disruption budget to make sure it is running and switching over when a node fails or gets upgraded. You want to either totally Oberprovision on memory or look into keep 2400 to make sure to fine-tune memory before k8s starts to throw your da…

> Use a helm chart and just bring your own little database for dev test and e2e tests. dev, test, and e2e tests should be done against full-size db clones

You think I'm going to clone a multiple petabyte database just to run some tests?

Re: Running Databases on Kubernetes

#20

Post by axlee »

samokhvalov wrote: post #11
Felminor wrote: post #5

That's just a really really bad write-up on the real problem on running a database on k8s. You need ha because k8s should run already with automatic node upgrades. You need a pod disruption budget to make sure it is running and switching over when a node fails or gets upgraded. You want to either totally Oberprovision on memory or look into keep 2400 to make sure to fine-tune memory before k8s starts to throw your da…

> Use a helm chart and just bring your own little database for dev test and e2e tests. dev, test, and e2e tests should be done against full-size db clones

> dev, test, and e2e tests should be done against full-size db clones

that's cute, what is your "full-size"? I don't have 2 days to run a test, and I'm pretty sure every single compliance requirements we are following would get obliterated the second someone hears about us doing that

Post reply on HN