Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Postgres in Kubernetes [pdf]

static.sched.com

Post reply on HN
Print view View original

21–30 of 101 posts

Re: Running Postgres in Kubernetes [pdf]

#21

Post by jeremychone »

Looks interesting but difficult to get the details from just the slides.

Also, not sure why Azure Arc still gets mentioned. I would have expected something more cloud independent.

Our approach, for now, is to use Kubernetes Postgres for dev, test, and even stage, but cloud Postgres for prod. We have one db.yaml that in production just become an endpoint so that all of the services do not even have to know if it is an internal or external Postgres.

Another interesting use of Kubernetes Postgres would be for some transient but bigger than memory store that needs to be queryable for a certain amount of time. It's probably a very niche use-case, but the deployment could be dramatically more straightforward since HA is not performance bound.

Re: Running Postgres in Kubernetes [pdf]

#22

Post by tyingq »

nightowl_games wrote: post #17

I think cockroachDB is designed for this.

They've thought about the use case. But it still ends up being a cluster inside a cluster, which sounds potentially pretty bad to me. Clusters of different types, mostly unaware of each other. Schema changes and database version upgrades would be complicated.

Re: Running Postgres in Kubernetes [pdf]

#23

Post by sasavilic »

Unless you have a really good shared storage, I don't see any advantage for running Postgres in Kubernetes. Everything is more complicated without any real benefit. You can't scale it up, you can't move pod. If pg fails to start for some reason, good luck jumping into container to inspect/debug stuff. I am neither going to upgrade PG every 2 weeks nor it is my fresh new microservice that needs to be restarted when it crashes or scaled up when I need more performances. And PG has high availability solution which kind of orthogonal to what k8s offers.

One could argue that for sake of consistency you could run PG in K8S, but that is just hammer & nail argument for me.

But if you have a really good shared storage, then it is worth considering. But, I still don't know if any network attached storage can beat local attached RAID of Solid state disks in terms of performance and/or latency. And there is/was fsync bug, which is terrible in combination with somewhat unreliable network storage.

For me, I see any database the same way I see etcd and other components of k8s masters: they are the backbone. And inside cluster I run my apps/microservices. This apps are subject to frequent change and upgrades and thus profit most from having automatic recovery, failover, (auto)scaling, etc.

Re: Running Postgres in Kubernetes [pdf]

#24

Post by perrygeo »

mamon wrote: post #4
aprdm wrote: post #2

That looks very interesting and super complex. I wonder how many companies really need this complexity, I bet 99.99% of the companies could get away with vertical scaling the writes and horizontal scaling the read only replica which would reduce the number of moving parts a lot. I have yet to play much with kubernetes but when I see those diagrams it just baffles me how people are OK with running so much complexity i…

It actually is not that complex. I'm using Crunchy Postgres Operator at my current employer. You get an Ansible playbook to install an Operator inside Kubernetes, and after that you get a commandline administration tool that let's you create a cluster with a simple pgo create cluster command. Most administrative tasks like creating or restoring backups (which can be automatically pushed to S3) are just one or two pgo…

Building systems on top of complexity doesn't shield anyone from it. The author acknowledges this explicitly:

> High Effort - Running anything in Kubernetes is complex, and databases are worse

By definition, it's more stuff you need to know.

Even if the K8s operator saves time for 95% of the use cases, the last 5% is required. For instance, how do these operators handle upgrading extensions that require on-disk changes? Can you upgrade them concurrently with major version PG upgrades? When the operator doesn't provide a command line admin tool that fits your needs, how do you proceed?

Re: Running Postgres in Kubernetes [pdf]

#25

Post by GordonS »

So, a bit OT, but I'm looking for some advice on building a Postgres cluster, and I'm pretty sure k8s is going to add a lot of complexity with no benefit.

I'm a Postgres fan, and use it a lot, but I've never actually used it in a clustered setup.

What I'm looking at clustering for is not really for scalability (still at the stage where we can scale vertically), but for high availability and backup - if one node is done for update, or crashes, the other node can take over, and I'd also ideally like point-in-time restore.

There seems to be a plethora of OSS projects claiming to help with this, so it looks like there isn't "one true way" - I'd love to hear how people are actually setting up their Postgres clusters for in practice?

Re: Running Postgres in Kubernetes [pdf]

#26

Post by random3 »

sasavilic wrote: post #23

Unless you have a really good shared storage, I don't see any advantage for running Postgres in Kubernetes. Everything is more complicated without any real benefit. You can't scale it up, you can't move pod. If pg fails to start for some reason, good luck jumping into container to inspect/debug stuff. I am neither going to upgrade PG every 2 weeks nor it is my fresh new microservice that needs to be restarted when it…

GCP volumes are over network already. You can deploy stateful workloads using StatefulSets. We've run an HBase workloads for development purposes (about 20-30x cheaper than BigTable) and it worked great (no issues for over 12 months). While Postgres is hardly a distributed database, there may be some advantages to ensure availability and perhaps even more in replicated setup.

Re: Running Postgres in Kubernetes [pdf]

#27

Post by random3 »

GordonS wrote: post #25

So, a bit OT, but I'm looking for some advice on building a Postgres cluster, and I'm pretty sure k8s is going to add a lot of complexity with no benefit. I'm a Postgres fan, and use it a lot, but I've never actually used it in a clustered setup. What I'm looking at clustering for is not really for scalability (still at the stage where we can scale vertically), but for high availability and backup - if one node is do…

The main advantage with Kubernetes (especially in low ops environments like GKE) is not scalability, but availability and ease of development (spinning things up and down is super-easy). The learning curve to stand something up is not very high and pays of over time compared to SSH-ing into VMs.

Re: Running Postgres in Kubernetes [pdf]

#28

Post by lhenk »

GordonS wrote: post #25

So, a bit OT, but I'm looking for some advice on building a Postgres cluster, and I'm pretty sure k8s is going to add a lot of complexity with no benefit. I'm a Postgres fan, and use it a lot, but I've never actually used it in a clustered setup. What I'm looking at clustering for is not really for scalability (still at the stage where we can scale vertically), but for high availability and backup - if one node is do…

Patroni might be interesting: https://github.com/zalando/patroni

Re: Running Postgres in Kubernetes [pdf]

#29

Post by rafiss »

tyingq wrote: post #22
nightowl_games wrote: post #17

I think cockroachDB is designed for this.

They've thought about the use case. But it still ends up being a cluster inside a cluster, which sounds potentially pretty bad to me. Clusters of different types, mostly unaware of each other. Schema changes and database version upgrades would be complicated.

There certainly are pain points. I don't work on this myself, but one of our other engineers wrote this blog post [0] that discusses the experience of running CockroachDB in k8s and why we chose to use it for our hosted cloud product. Another complication mentioned in there is about how to deal with the multi-region case.

[0] https://www.cockroachlabs.com/blog/managed-cockroachdb-on-ku...

Re: Running Postgres in Kubernetes [pdf]

#30

Post by caiobegotti »

Please don't. It's not because it's possible that it is a good idea. The PDF itself clearly shows how it can get complex quickly. The great majority of people won't ever be able to do this properly, securely and with decent reliability. Of course I may have to swallow my words in the future in case a job requires it but unless you REALLY REALLY REALLY need PostgreSQL inside Kubernetes IMHO you should just stick with private RDS or Cloud SQL then point your Kubernetes workloads to it inside your VPCs, all peered etc. Your SRE mental health, your managers and company costs will thank you.
Post reply on HN