Instead of messing around with Kubernetes, I would rather advocate for something like Amazon RDS.
Running Postgres in Kubernetes [pdf]
91–100 of 101 posts
Re: Running Postgres in Kubernetes [pdf]
#92Earlier quoted context omitted.
IOPS limitations and network latency are the reason I want to run my Postgres replicas in k8s. Every machine gets NVMe and a replica instance. They're still cattle, but they can serve a lot of requests. Database CPU usage is negligible. It was easier to put a replica on every host than try to rearchitect things to tolerate single-digit millisecond RTT to a cloud database.
One thing I dislike about SQL databases is the synchronous protocol. It really feels like I should be able to send a batch of queries all at once and then process the response in bulk instead of waiting for each individual query to respond incurring a round trip every single time. You can obviously rewrite your queries so that they fetch everything upfront in one single huge query but that is a lot of work and the en…
Re: Running Postgres in Kubernetes [pdf]
#93Earlier quoted context omitted.
I agree that the k8s ecosystem isn't quite as complex as it seems at first, but specifically running stateful apps does come pretty close to earning the bad reputation. (Disclaimer: I've tried and failed several times to get pgsql up and running in k8s with and without operators, so that either makes me unqualified to discuss this, or perfectly qualified to discuss this) If the operator were simple enough to be insta…
I run multi terabyte, billions of rows HA postgres in kubernetes using a helm chart and Patroni (baked into the chart), which uses the native k8s API for automatic failover and pgbackrest for seamlessly provisioning new replicas. It's a single helm chart and is by far the easiest DBA I've ever done in many years.
I realise this is possibly asking you to give away secret sauce, but is this written up anywhere? Having an example to point at to be able to say "Look, this isn't scary, we can contemplate retiring that nasty lump of tin underneath that Oracle instance after all" would be quite a valuable contribution.
Re: Running Postgres in Kubernetes [pdf]
#94Google Cloud blog, gently dissuading you from running a traditional DB in K8s: https://cloud.google.com/blog/products/databases/to-run-or-n... K8s docs explaining how to run MySQL: https://kubernetes.io/docs/tasks/run-application/run-replica... You could also run it with Nomad, and skip a few layers of complexity: https://learn.hashicorp.com/nomad/stateful-workloads/host-vo... / https://mysqlrelease.com/2017/12/hashi…
> One of the big problems of K8s is it's a monolith while I pretty much agree with everything else you mention, I think it's kind of the opposite; since k8s is fundamentally an API, it's very modular and extensible and this is why it's being successful (I agree it wants you to do things its way and things like databases need to be shimmed at the moment, so the conclusion is similar... for now)
It looks like microservices from the high level. But then you dig into each component, and realize that other than using APIs, they still require all the other components, use a shared storage layer, sometimes use non-standard protocols. The kube-controller-manager alone is literally a tiny monolith: a single binary with 5 different controllers in it. K8s operates like a monolith because you mostly can't just remove one layer and have it keep running.
Compare that to HashiCorp's tools. You can run a K8s-like system composed mostly of Hashi's tools, but you can also run each of those tools as a single complete unit of just itself. Now, each of those tools is actually multiple components in one, like mini-monoliths. But in operation, they can work together or as independent services. The practical result is truly stand-alone yet reusable and interoperable components. That's the kind of DevOps methodology I like: buy a wheelbarrow today and hitch it to an ATV next week, rather than going out and buying a combine harvester before you need it.
Re: Running Postgres in Kubernetes [pdf]
#95Earlier quoted context omitted.
> One of the big problems of K8s is it's a monolith while I pretty much agree with everything else you mention, I think it's kind of the opposite; since k8s is fundamentally an API, it's very modular and extensible and this is why it's being successful (I agree it wants you to do things its way and things like databases need to be shimmed at the moment, so the conclusion is similar... for now)
It looks like microservices from the high level. But then you dig into each component, and realize that other than using APIs, they still require all the other components, use a shared storage layer, sometimes use non-standard protocols. The kube-controller-manager alone is literally a tiny monolith: a single binary with 5 different controllers in it. K8s operates like a monolith because you mostly can't just remove…
Yes, these are very good points, all true, thanks.
I still think that while this monolith has its drawbacks, the fact that any component can be substituted as long as it confirms to the official API is really powerful. For example k3s uses sqlite instead of etcd.
Having small components that do one thing well (Unix philosophy) is certainly one way to go (I still haven't found somebody who doesn't love Hashicorp tooling, myself included) but the k8s idea of having one (big, possibly bloated for many cases) "standard" way of doing things while being customizable/extensible is really powerful. If Hashi came up with some (extendible) glue/package tooling then a lot of people doing/looking at k8s right now will seriously look at them (myself included).
Re: Running Postgres in Kubernetes [pdf]
#96Earlier quoted context omitted.
It actually is not that complex. I'm using Crunchy Postgres Operator at my current employer. You get an Ansible playbook to install an Operator inside Kubernetes, and after that you get a commandline administration tool that let's you create a cluster with a simple pgo create cluster command. Most administrative tasks like creating or restoring backups (which can be automatically pushed to S3) are just one or two pgo…
I agree that the k8s ecosystem isn't quite as complex as it seems at first, but specifically running stateful apps does come pretty close to earning the bad reputation. (Disclaimer: I've tried and failed several times to get pgsql up and running in k8s with and without operators, so that either makes me unqualified to discuss this, or perfectly qualified to discuss this) If the operator were simple enough to be insta…
agreed re: configuring the operator. cockroach labs (full disclosure: I'm an employee) are building a HA pgsql alternative that Just Works with k8s to solve exactly this problem: https://www.cockroachlabs.com/blog/kubernetes-orchestrate-sq...
Re: Running Postgres in Kubernetes [pdf]
#97Earlier quoted context omitted.
I agree that the k8s ecosystem isn't quite as complex as it seems at first, but specifically running stateful apps does come pretty close to earning the bad reputation. (Disclaimer: I've tried and failed several times to get pgsql up and running in k8s with and without operators, so that either makes me unqualified to discuss this, or perfectly qualified to discuss this) If the operator were simple enough to be insta…
agreed re: configuring the operator. cockroach labs (full disclosure: I'm an employee) are building a HA pgsql alternative that Just Works with k8s to solve exactly this problem: https://www.cockroachlabs.com/blog/kubernetes-orchestrate-sq...
spencer kimball and alex polvi deploy a scalable stateful application on cockroachdb and k8s in 3 min: https://www.youtube.com/watch?v=PIePIsskhrw
Re: Running Postgres in Kubernetes [pdf]
#98Earlier quoted context omitted.
You don't run shared/network storage. You run PVCs on local storage and you run an HA setup. You ship log files every 10s/30s/1m to object storage. You test your backups regularly which k8s is great for. All of this means that I don't worry about most things you mention. PG upgrade? Failover and upgrade the pod. Upgrade fails? wal-g clone from object storage and rejoin cluster. Scaling up? Adjust the resource claims.…
It sounds pretty simple and enticing. Any problems convincing the team of this route?
Historically speaking, convincing anyone to run data stores on k8s is a giant pain. This isn't helped by all the mythology around "k8s is great for stateless" further promoted by folks like Kelsey Hightower who firmly believes that one should rely on the cloud provider's data stores.
This is great for the cloud provider as those are high margin services, I for one would rather have a single orchestration API that I have to interact with, that being the one offered by k8s. All the benefits of running workloads in k8s apply equally to data stores. Cattle, not pets. If your data store needs coddling then it's a pet and you're doing it wrong.
Re: Running Postgres in Kubernetes [pdf]
#99Unless you have a really good shared storage, I don't see any advantage for running Postgres in Kubernetes. Everything is more complicated without any real benefit. You can't scale it up, you can't move pod. If pg fails to start for some reason, good luck jumping into container to inspect/debug stuff. I am neither going to upgrade PG every 2 weeks nor it is my fresh new microservice that needs to be restarted when it…
The nice thing about running a DB inside a cluster is running your entire application, end to end, through one unified declarative model. It's really really easy to spin up a brand new dev or staging environment. generally though in production, you're not going to be taking down DBs on purpose. If it's not supposed to be ephemeral, it doesn't fit the model
In my previous gig (2017 timeframe), we moved from AWS instances with terraform plus chef which took a couple hours of focused engineer time to generate a new dev environment. This was after some degree of automation, but there were still a great deal of steps one had to get through and if you wanted to automate all that how would you glue it all together? Bash scripts?
We transition to k8s, with PG and other data stores in cluster, specifically RabbitMQ, and Mongo, which runs surprisingly well in k8s. In any case, after the whole adoption period and a great deal of automation work against the k8s APIs, we were able to get new dev environment provisioning down to 90 seconds.
There was clearly some pent up demand for development resources as we went from a few dev environments to roughly 30 in one month's time.
Following that, the team added the ability to "clone" any environment including production ones, that is, the whole data set and configuration was replicated into a new environment. One could also replicate data streaming into this new environment, essentially having an identical instance of a production service with incoming data.
This was a huge benefit for development and testing and further drove demand for environments. If a customer had a bug or an issue, a developer could fire up a new environment with a fix branch, test the fix on the same data and config, and then commit that back to master making its way into production.
These are the benefits of running data stores governed by one's primary orchestration framework. Sounds less threatening when put that way, eh?
Re: Running Postgres in Kubernetes [pdf]
#100Unless you have a really good shared storage, I don't see any advantage for running Postgres in Kubernetes. Everything is more complicated without any real benefit. You can't scale it up, you can't move pod. If pg fails to start for some reason, good luck jumping into container to inspect/debug stuff. I am neither going to upgrade PG every 2 weeks nor it is my fresh new microservice that needs to be restarted when it…
You don't run shared/network storage. You run PVCs on local storage and you run an HA setup. You ship log files every 10s/30s/1m to object storage. You test your backups regularly which k8s is great for. All of this means that I don't worry about most things you mention. PG upgrade? Failover and upgrade the pod. Upgrade fails? wal-g clone from object storage and rejoin cluster. Scaling up? Adjust the resource claims.…
For dummys like me who are interested but not familiar with infra.
PVC = Persistent Volume Claim HA = High avalability PG = Postgres