41–50 of 87 posts
Shameless plug: if you want to start running your own PostgreSQL with SSL, SELinux, automatic system updates, attached storage, and more... I have a simple demo as part of https://deploymentfromscratch.com/.
Best way to run database is in static pods. You'll get all the benefit of kubernetes ecosystem (monitoring, logs, inventory, access control) without any drawbacks.
You mean without a deployment or statefulset? How would you connect to the DB without a service providing a persistent DNS name?
these manifests can be targeted with a Service.
However my preferred solution for databases inside k8s is still zalando-operator with spilo and local storage. it's rock solid. I once used kubedb and lost the database because of bugs, so I'm staying away from them. Crunchy looks solid aswell but it looks like it's a little bit more polished but harder to configure than zalando's solution which is more built to their needs.
Earlier quoted context omitted.
Because kubernetes is a fantastic platform that provides a lot of node and app management functionality out of the box. Stateful services like databases are not really a challenge anymore given recent developments in both K8S and database containerization.
With a database I generally want to have it on its own dedicated host with fast NVME drives, memory, cpu etc, maybe some OS specific system level configuration options for performance etc. I don't want to binpack it on a cluster of nodes each contending for system resources. Also RDBMS databases are not really designed to be ephemeral/elastic, they are stateful/persistent by nature. What does Kubernetes offer in this…
What you want to run, and how, where, under what conditions, etc. is all specified in declarative files. Like I said, stateful services are not a problem anymore. K8S can deploy a database service as instances exclusive to specific nodes with the appropriate storage and networking config.
It's like an invisible assistant to automate what you would be doing yourself, or using several other tools to do anyway. A well-tested way to handle most of the common infrastructure issues for any deployed software in a single platform.
EDIT: I'll add that running K8S itself can be challenging, but that's because complexity can't be erased but only abstracted. I recommend using managed K8S to offload this.
Earlier quoted context omitted.
At least in my case, because your ops team has decided that everything that gets deployed, gets deployed in k8s (unless you have a Very! good reason not to)
Kubernetes is great but not everything fits in kubernetes, this is a bad ops team if this is the case.
I think this is a good ops team mantra. PostgreSQL unless you have a very good reason otherwise, and k8s unless you have a very good reason otherwise.
Decisions like this allow for economies of scale with tooling, monitoring/observability, etc.
Saying no to snowflakes is an important part of having a good ops team.
Earlier quoted context omitted.
With a database I generally want to have it on its own dedicated host with fast NVME drives, memory, cpu etc, maybe some OS specific system level configuration options for performance etc. I don't want to binpack it on a cluster of nodes each contending for system resources. Also RDBMS databases are not really designed to be ephemeral/elastic, they are stateful/persistent by nature. What does Kubernetes offer in this…
Say you have 100 database clusters and you want to schedule them on dedicated servers. A kubernetes node can advertise those local block devices as persistent volumes. Your StatefulSet can request that class of local storage and the kubernetes scheduler will then find a volume for the pods on a node that the pod can fit on. You then get all the features of kubernetes operators plus the performance of dedicated nodes…
Actually no. Currently you need to manually delete the persistent volume claim. Followed by deleting the pod. For the statefulsets ti to create a new PVC.
There even was a race condition until like 1.21 that made this operation fail 20% of the time so you'd have to delete the pod twice or the system would deadlock.
It'd be nice if the PVC would claim a new PVC if the orchestration layer knows 100% the PV is gone but it currently doesn't do this yet.
It's not an enormous problem once you're aware of it. But it does mean things don't auto heal at all. A node going down with local storage on it needs manual intervention to get the system running and healthy again.
There might be some operators that automate this away but i haven't seen it yet in the wild.
Earlier quoted context omitted.
Because kubernetes is a fantastic platform that provides a lot of node and app management functionality out of the box. Stateful services like databases are not really a challenge anymore given recent developments in both K8S and database containerization.
> Stateful services like databases are not really a challenge anymore I've faced so many issues with the CSI driver which doesn't attach/deattach properly when you modify anything on the Statetfulset. Additionally, you need to pin your DBs ("statefulset workloads") to certain Nodes based on the AZs since a block storage like an EBS cannot really be attached from Zone 1 to Zone 2 automatically. Based on these limitati…
There used to be bugs with this years ago but all storage providers now get this right.
Earlier quoted context omitted.
if you run your stateless services on an orchestration system like k8s, you have two options: a) run your stateful services (such as Postgres) on the same orchestration system b) choose a second orchestration system (often a more "traditional" infrastructure provisioning system like Chef/Puppet/Ansible/SaltStack) that will only manage your stateful services, and deal with the care & feeding & mental overhead of havin…
If you choose a) you don't really get to avoid b), since something still has to bootstrap your Kubernetes nodes themselves. So it's more a question whether you'd rather write a Kubernetes manifest or a Salt/Ansible config; and with the complexity and limitations of a lot of storage operators, and no real "orchestration" benefit once you have pinned deployments to particular nodes anyway, sometimes it's the latter. Th…
Not quite true if you use a managed K8S offering - and it seems a self-managed cluster is something to be avoided until absolutely necessary. Even with a managed cluster like GKE there's plenty of operational overhead to keep everything up-to-date and working properly.
Earlier quoted context omitted.
StatefulSets, Node Groups, and Node Affinity/Taints solve some of those issues.
A database is not a service well suited for ephemeral, quick scale down and scale up container workloads that K8s really excells at, and if you have nodes specifically configured to run databases and nothing else then you should simply run them there directly and not mangle K8s to replicate what will essentially be equivalent to running them there directly.
Being able to get essentially the same result, but with our familiar processes, tooling, infra, that was really helpful getting things off the ground quickly and keeping them stable. We picked k8s/GKE way before that as a platform where a small set of abstractions allow us to do a lot of different things, and that makes it easy to debug unfamiliar workloads, keeping operations overhead low, and it worked out really well, and much easier for people with little ops experience to learn and get to a point where they could take care of their workloads themselves.
And it never felt we had to mangle k8s for our stateful workloads at all. Statefulsets, storage, node affinity etc. feel very much like organic building blocks and work really really well, with very few surprises. I guess k8s has come a long way as far as stateful workloads are concerned.
Earlier quoted context omitted.
At least in my case, because your ops team has decided that everything that gets deployed, gets deployed in k8s (unless you have a Very! good reason not to)
Kubernetes is great but not everything fits in kubernetes, this is a bad ops team if this is the case.