Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Postgres in Kubernetes [pdf]

static.sched.com

Post reply on HN
Print view View original

11–20 of 101 posts

Re: Running Postgres in Kubernetes [pdf]

#11

Post by LoSboccacc »

adamcharnock wrote: post #8
aprdm wrote: post #2

That looks very interesting and super complex. I wonder how many companies really need this complexity, I bet 99.99% of the companies could get away with vertical scaling the writes and horizontal scaling the read only replica which would reduce the number of moving parts a lot. I have yet to play much with kubernetes but when I see those diagrams it just baffles me how people are OK with running so much complexity i…

I generally work with smaller companies, but early on (Kubernetes 1.4 ish) I found that hosting mission-critical stateful services inside Kubernetes was more trouble than it was worth. I now run stand-alone Postgres instances in which each service has its own DB. I’ve found this very reliable. That being said, I think Kubernetes now has much better support for this kind of thing. But given my method has been so stabl…

> stateful services

yeah either these services support natively partitioning, fail over and self recovery or you have to be extremely careful not to cause any eviction or agent crash ever.

even something born for the cloud like cockroachdb can fail in interesting ways if the load order varies and you can't just autoscale it because every new node has to be nudged into action with a manual cluster-init, and draining nodes after the peak means manually telling the cluster not to wait for the node to come alive ever again for each node, wait for the repartitioning and then repeat for as many nodes as you need to scale back

Re: Running Postgres in Kubernetes [pdf]

#12

Post by blyry »

ooh! I've been running the Zalando operator in production on Azure for ~ a year now, nothing crazy but a couple thousand qps and a tb of data spread across a several clusters. It's been a little rough since it was designed for AWS, but pretty fun. At this point, I'm 50/50, our team is small and i'm not sure that the extra complexity added by k8s solved any problems that azures managed postgres product doesn't also solve. We weren't sure we were going to stay on azure at the time we made the decision as well -- if I was running in a hybrid cloud environment I would 100% choose postgres on k8s.

The operator let us ramp up real quickly with postgres as a POC and gave us mature clustering and point-in-time restoration, and the value is 100% there for dev/test/uat instances, but depending on our team growth it might be worth it to switch to managed for some subset of those clusters once "Logical Decoding" goes GA on the azure side. Their hyperscale option looks pretty fun as well, hopefully some day i'll have that much data to play with.

I can also say that the Zalando crew has been crazy responsive on their github, it's an extremely well managed open source project!

Re: Running Postgres in Kubernetes [pdf]

#13

Post by dewey »

mamon wrote: post #4
aprdm wrote: post #2

That looks very interesting and super complex. I wonder how many companies really need this complexity, I bet 99.99% of the companies could get away with vertical scaling the writes and horizontal scaling the read only replica which would reduce the number of moving parts a lot. I have yet to play much with kubernetes but when I see those diagrams it just baffles me how people are OK with running so much complexity i…

It actually is not that complex. I'm using Crunchy Postgres Operator at my current employer. You get an Ansible playbook to install an Operator inside Kubernetes, and after that you get a commandline administration tool that let's you create a cluster with a simple pgo create cluster command. Most administrative tasks like creating or restoring backups (which can be automatically pushed to S3) are just one or two pgo…

Usually things become complex once something isn't going as planned. Like if your database slows down because the pods get scheduled on a weird node with some noisy neighbour, your backups failed because the node went down or other more hidden issues that take a lot longer to debug compared to some Postgres running on a normal compute instance somewhere.

It's just additional layers to dig through if something goes wrong, if everything works even the most complex systems are nice to operate so I wouldn't call it less complex just because someone wrote a nice wrapper for the happy path.

Re: Running Postgres in Kubernetes [pdf]

#14

Post by jashmatthews »

mamon wrote: post #4
aprdm wrote: post #2

That looks very interesting and super complex. I wonder how many companies really need this complexity, I bet 99.99% of the companies could get away with vertical scaling the writes and horizontal scaling the read only replica which would reduce the number of moving parts a lot. I have yet to play much with kubernetes but when I see those diagrams it just baffles me how people are OK with running so much complexity i…

It actually is not that complex. I'm using Crunchy Postgres Operator at my current employer. You get an Ansible playbook to install an Operator inside Kubernetes, and after that you get a commandline administration tool that let's you create a cluster with a simple pgo create cluster command. Most administrative tasks like creating or restoring backups (which can be automatically pushed to S3) are just one or two pgo…

Crunchy PGO is super cool but I'm not sure how we got to the idea that it's not that complex compared to a managed service like RDS.

Re: Running Postgres in Kubernetes [pdf]

#15

Post by merb »

aprdm wrote: post #2

That looks very interesting and super complex. I wonder how many companies really need this complexity, I bet 99.99% of the companies could get away with vertical scaling the writes and horizontal scaling the read only replica which would reduce the number of moving parts a lot. I have yet to play much with kubernetes but when I see those diagrams it just baffles me how people are OK with running so much complexity i…

btw. zalando operator is more rough, but still pretty easy to use. crunchy operator does not work in every environment but is extremly simple (btw. the crunchy operator uses the building blocks of zalando) used zalando operator since k8s 1.4, no data loss, everything just works, ok major upgrades are rough, but they are rough even without zalando operator.

Re: Running Postgres in Kubernetes [pdf]

#16

Post by Townley »

mamon wrote: post #4
aprdm wrote: post #2

That looks very interesting and super complex. I wonder how many companies really need this complexity, I bet 99.99% of the companies could get away with vertical scaling the writes and horizontal scaling the read only replica which would reduce the number of moving parts a lot. I have yet to play much with kubernetes but when I see those diagrams it just baffles me how people are OK with running so much complexity i…

It actually is not that complex. I'm using Crunchy Postgres Operator at my current employer. You get an Ansible playbook to install an Operator inside Kubernetes, and after that you get a commandline administration tool that let's you create a cluster with a simple pgo create cluster command. Most administrative tasks like creating or restoring backups (which can be automatically pushed to S3) are just one or two pgo…

I agree that the k8s ecosystem isn't quite as complex as it seems at first, but specifically running stateful apps does come pretty close to earning the bad reputation.

(Disclaimer: I've tried and failed several times to get pgsql up and running in k8s with and without operators, so that either makes me unqualified to discuss this, or perfectly qualified to discuss this)

If the operator were simple enough to be installed/uninstalled via a helm chart that Just Worked, I'd feel better about the complexity. But running a complicated, non-deterministic ansible playbook scares me. The other options (installing a pgo installer, or installing an installer to your cluster) are no better.

Also, configuring the operator is more complicated than it should be. Devs and sysadmins alike are used to `brew install postgresql-server` or `apt install postgresql-server` working just fine for 99% of use cases. I'll grant that it's not apples-to-apples since HA pgsql has never been easy, but if the sales pitch is that any superpower-less k8s admin can now run postgres, I think the manual should be shorter.

Re: Running Postgres in Kubernetes [pdf]

#18

Post by craigkerstiens »

mamon wrote: post #4
aprdm wrote: post #2

That looks very interesting and super complex. I wonder how many companies really need this complexity, I bet 99.99% of the companies could get away with vertical scaling the writes and horizontal scaling the read only replica which would reduce the number of moving parts a lot. I have yet to play much with kubernetes but when I see those diagrams it just baffles me how people are OK with running so much complexity i…

It actually is not that complex. I'm using Crunchy Postgres Operator at my current employer. You get an Ansible playbook to install an Operator inside Kubernetes, and after that you get a commandline administration tool that let's you create a cluster with a simple pgo create cluster command. Most administrative tasks like creating or restoring backups (which can be automatically pushed to S3) are just one or two pgo…

Not the author of the slides, but know him well. A number of things to chime in on. First thanks for the kinda words on the Crunchy operator.

Second, on the earlier question higher in the thread about why would you choose to run a database in K8s. In my experience and what I've observed it's not so much you choose explicitly to run a database in K8s. Instead you've decided on K8s as your orchestration layer for a lot of workloads and it's become your standardized mechanism for deploying and managing apps. In some sense it's more that it's the standard deployment mechanism than anything else.

If you're running and managing a single Postgres database and don't have any K8s anywhere setup, I can't say I'd recommend going all in on K8s just for that. That said if you are using it then going with one of the existing operators is going to save you a lot.

Re: Running Postgres in Kubernetes [pdf]

#19

Post by craigkerstiens »

jashmatthews wrote: post #14
mamon wrote: post #4

Earlier quoted context omitted.

It actually is not that complex. I'm using Crunchy Postgres Operator at my current employer. You get an Ansible playbook to install an Operator inside Kubernetes, and after that you get a commandline administration tool that let's you create a cluster with a simple pgo create cluster command. Most administrative tasks like creating or restoring backups (which can be automatically pushed to S3) are just one or two pgo…

Crunchy PGO is super cool but I'm not sure how we got to the idea that it's not that complex compared to a managed service like RDS.

Coming from someone at Crunchy I don't disagree on the notion of managed service being easier than running/managing yourself inside Kubernetes. Clicking a button and having things taken care of you is great.

Though personally I do feel like much of the managed services have not evolved/changed/improved since their inception many years ago. There is definitely some opportunity here to innovate, though that's probably not actually coupled with running it in K8s itself.

Post reply on HN