41–50 of 78 posts
I guess there must be a usecase in missing here, but RDS is working so well for me, it's hard to imagine why I would not shift most of the operational concerns to this competent vendor. The only thing I can think of is cost. My usage probably isn't high enough where there is any financial benefit to an alternative... but if it was, maybe I'd be considering this.
Earlier quoted context omitted.
> Use a helm chart and just bring your own little database for dev test and e2e tests. dev, test, and e2e tests should be done against full-size db clones
> dev, test, and e2e tests should be done against full-size db clones that's cute, what is your "full-size"? I don't have 2 days to run a test, and I'm pretty sure every single compliance requirements we are following would get obliterated the second someone hears about us doing that
That's just a really really bad write-up on the real problem on running a database on k8s. You need ha because k8s should run already with automatic node upgrades. You need a pod disruption budget to make sure it is running and switching over when a node fails or gets upgraded. You want to either totally Oberprovision on memory or look into keep 2400 to make sure to fine-tune memory before k8s starts to throw your da…
This is difficult to impossible to do with databases; even if your database has a built-in recovery method for when a primary is taken offline, in such a way that allows for zero-downtime in theory, the reality is that such mechanisms depend on the secondary staying online until the failover mechanism is complete. If you turn over control of node upgrades to the cluster provider, the node under the secondary will get rebooted in the middle of the failover process, and you will get downtime at best, data loss at worst. What kubernetes teaches us is that databases aren't tied to the literal VM they're running on (which is now cattle), but rather on the availability of that node. If you run databases on kubernetes, you need to have a mechanism to slow down node upgrades.
Source: helped run hundreds of Elasticsearch and Kafka nodes on kubernetes in production at one point in my career
I guess there must be a usecase in missing here, but RDS is working so well for me, it's hard to imagine why I would not shift most of the operational concerns to this competent vendor. The only thing I can think of is cost. My usage probably isn't high enough where there is any financial benefit to an alternative... but if it was, maybe I'd be considering this.
When you start to have dozens or hundreds of databases in production, and developers asking "I need Postgres in production, why can't I just click a button and get a Postgres instance for my service in production?" then scaling the monitoring and firewalling gets a little more complicated. Hooking into standard Kubernetes monitoring and service meshes can really help to simplify things.
Earlier quoted context omitted.
> dev, test, and e2e tests should be done against full-size db clones that's cute, what is your "full-size"? I don't have 2 days to run a test, and I'm pretty sure every single compliance requirements we are following would get obliterated the second someone hears about us doing that
You have a tool to keep the structure of data but anonymize it, I’ve seen this a few times in healthcare regulated systems
I don't think the upsides are worth all the work. You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2? Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs.…
yeah but if your org has orchestration tooling built around k8s, in a way it becomes much easier to provision a DB with k8s, setup the service, routing, networking, roles, etc than it would be in terraform. especially if you have to repeat this process in like 20 envs (stage, prod) x multiple regions
Where I was the tooling was very focused on disposable api servers.
That's just a really really bad write-up on the real problem on running a database on k8s. You need ha because k8s should run already with automatic node upgrades. You need a pod disruption budget to make sure it is running and switching over when a node fails or gets upgraded. You want to either totally Oberprovision on memory or look into keep 2400 to make sure to fine-tune memory before k8s starts to throw your da…
> k8s should run already with automatic node upgrades This is difficult to impossible to do with databases; even if your database has a built-in recovery method for when a primary is taken offline, in such a way that allows for zero-downtime in theory, the reality is that such mechanisms depend on the secondary staying online until the failover mechanism is complete. If you turn over control of node upgrades to the c…
I've done it with Cassandra...and yeah Kafka can do it I've heard.
But those can be 30 hour operations even with you ducks in a row, and you better have backup strategies ready.
Fun story, Amazon said rds would be always be zero downtime upgrades. But then came a major version upgrade and .... Surprise it wasn't.
I don't think the upsides are worth all the work. You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2? Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs.…
That's just a really really bad write-up on the real problem on running a database on k8s. You need ha because k8s should run already with automatic node upgrades. You need a pod disruption budget to make sure it is running and switching over when a node fails or gets upgraded. You want to either totally Oberprovision on memory or look into keep 2400 to make sure to fine-tune memory before k8s starts to throw your da…
> k8s should run already with automatic node upgrades This is difficult to impossible to do with databases; even if your database has a built-in recovery method for when a primary is taken offline, in such a way that allows for zero-downtime in theory, the reality is that such mechanisms depend on the secondary staying online until the failover mechanism is complete. If you turn over control of node upgrades to the c…
Cloudnative PG rebuilds the secondary during failiover from the streamed WAL to an S3 endpoint. No primary needed.