Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-hosting a high-availability Postgres cluster on Kubernetes

ryan-schachte.com

Post reply on HN
Print view View original

41–50 of 92 posts

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#41

Post by Szpadel »

x86hacker1010 wrote: post #3
andrewmunsell wrote: post #2

My holiday project was doing another pass at my Homelab Kubernetes cluster, part of which involved switching to a proper operator to manage Postgres. Coincidentally, I setup cloudnative-pg ( https://github.com/cloudnative-pg/cloudnative-pg ) yesterday.

Any reason you landed on that Operator compared to what OP is using (Zalando)?

I was setting fairly important database with Zalando pg operator and after first good impressions it went downhill. after like a month of use WAL files used for point in time recovery started failing to offload to dedicated nodes and kept growing on database pods filling up all the space. I firstly assumed that maybe there is not enough space for some scheduled work (I do not really know details how this process work, I assumed that operator should handle all implementation details for me) but even after upscaling database 2.5x it just kept failing with full storage and requiring manual recovery to bigger storage, where most of it was WAL files.

HA didn't handled this case at all whole cluster went in crash loop

there was also issue of huge pages caused crashing and not easy way to disable those without some dirty injecting of config files at runtime

there could be some my fault at misconfiguration on by side, but I wasn't able to figure anything better from docs

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#42

Post by marcosdumay »

rad_gruchalski wrote: post #38
iamgopal wrote: post #20

Earlier quoted context omitted.

Me who have never used kubernetes, what if node crash ? Will I lost everything ?

The answer is: it depends. It depends on if you use persistent volumes, and how well is your pv isolated from the failed node. If done right, no data loss.

A big emphasis to the "if done right" part.

You should test your setup, because it's very often not done right, and it's easy to overlook a problem.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#44

Post by snapetom »

gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

I've never been a fan of deploying DBs, either traditional RDBMS or distributed, under containerization/orchestration other than quickly spinning up dev and test environments. Certainly not for production. Databases have been built for high availability for many decades. Sure, maybe RDBMS doesn't scale quite as easily as a node API application, but it's still not really rocket science and you usually have some leeway if you get a spike.

Adding Swarm or K8 just seems like redundant and unnecessary complexity.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#45

Post by debarshri »

williamdclt wrote: post #11
renegade-otter wrote: post #5

Earlier quoted context omitted.

This kind of resilience is a form of art, and it's also kind of a full-time job. I would not advise trying this for a "side project at work". Generally we all agree that we move to the cloud and it's "fully managed". If it goes down - that's the price we pay. If you have to ask the question "what if the RDS goes down", then you are really in a different universe. That last guarantee of uptime requires a ton of work,…

> If you have to ask the question "what if the RDS goes down", then you are really in a different universe. It does go down though, don’t neglect the possibility because it likely will happen. With very average workloads, I’ve seen RDS databases restart unexpectedly, read replicas being completely out of service, and even databases being completely frozen (can’t even connect as root). I’d still go with managed, but i…

Problem is that RDS comes at a price. It is purely about operation cost.

When you have 1500+ databases these cost add up. At that point, this kind of techniques are required to self host the databases. Price per DB with HPA na VPA is way lower than what you would pay for managed databases as well as you can hire a full time devops+dbadmin and still be cheaper.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#46

Post by seanhunter »

marcosdumay wrote: post #36
jauntywundrkind wrote: post #21

Earlier quoted context omitted.

Is the complexity better or worse than alternatives? What are the alternatives? People use complexity as a boogieman to justify throwing together their own really wild chaotic & only so-so tested "simple" alternatives all the time . To me, this feels like a modern wonder. We have layers of responsibility. Many people operate Kubernetes clusters already for all kinds of reasons. It provides a powerful broad base. Now…

> What are the alternatives? Installing the Postgres on the computer, or on a VM. Really, just the fact that some people keep asking this question is enough to question everything else they say. The alternatives are obvious .

Exactly. The way people successfully did HA and scalability on databases for more than 20 years before Kubernetes existed.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#47

Post by steveBK123 »

snapetom wrote: post #44
gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

I've never been a fan of deploying DBs, either traditional RDBMS or distributed, under containerization/orchestration other than quickly spinning up dev and test environments. Certainly not for production. Databases have been built for high availability for many decades. Sure, maybe RDBMS doesn't scale quite as easily as a node API application, but it's still not really rocket science and you usually have some leeway…

Wouldn't most data driven, disk-IO intensive workloads be a bad fit for k8s? Especially when its not purely read-only..

That seems the crux of it.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#48

Post by snapetom »

mikepurvis wrote: post #35
levkk wrote: post #17

Earlier quoted context omitted.

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

"K8s is not really for stateful systems" As a relative novice in the space, I'm grateful to hear someone say this out loud. K8s seems perfect to me for quickly scaling transient stuff like pipeline workers, web servers, but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications, dat…

Our company is finally looking at containerization and orchestration, and one product had the gall to say, Docker isn't for us. The non-technical people gasped because all the other products are moving towards orchestration!

Why? it's an ancient Windows Client/Server app. Each "node" manages its own state and communicates with each other in this proprietary, janky-ass way. It takes 10 minutes to start up a node.

K8/Swarm isn't going to do squat for this team except maybe launch dev/test environments a little easier.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#49

Post by DasIch »

levkk wrote: post #17
gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

Here is an announcement from last year stating that volume expansion is stable since 1.24: https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/

The zalando postgres-operator also mentions as a feature:

> Live volume resize without pod restarts (AWS EBS, PVC)

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#50

Post by snapetom »

steveBK123 wrote: post #47
snapetom wrote: post #44

Earlier quoted context omitted.

I've never been a fan of deploying DBs, either traditional RDBMS or distributed, under containerization/orchestration other than quickly spinning up dev and test environments. Certainly not for production. Databases have been built for high availability for many decades. Sure, maybe RDBMS doesn't scale quite as easily as a node API application, but it's still not really rocket science and you usually have some leeway…

Wouldn't most data driven, disk-IO intensive workloads be a bad fit for k8s? Especially when its not purely read-only.. That seems the crux of it.

Absolutely. I haven't gotten in the weeds of K8 volumes, but certainly have done a lot with Docker volumes including adding some PRs to fix some issues I've ran into. There is a lot under the hood with Docker volumes. With how DBs like to optimize their IO operations, I'm frankly a little amazed how it somehow mostly works between the DB -> orchestrator volumes -> file system.
Post reply on HN