Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-hosting a high-availability Postgres cluster on Kubernetes

ryan-schachte.com

Post reply on HN
Print view View original

31–40 of 92 posts

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#31

Post by __MatrixMan__ »

ssijak wrote: post #10

Is Kubernetes still hard in 2024?

I can only speak from the perspective of somebody trying to manipulate k8s environments for use in test, but I'd say yes.

Four times across three companies I have run across frameworks which set everything up out of band and then ask the tests to test it. Maybe these are shell scripts written by the k8siest person on the team. Maybe they're something like tilt. Whatever the case they're always a black box to the majority of the people who are writing application code.

They get you 90% there, but eventually somebody wants assurances that some environment variable has the desired effect, and suddenly you need to penetrate that black box and change it so that there are multiple kinds of "up" and the right tests run against each state.

K8s tooling is commonly installed via curl, so once you unravel the black box and integrate it with your tests you end up with a lot of fragile interfaces to things like kustomize, kubectl, kind... Fragile because maybe the other dev has a different version installed. Nix dev shells solve this, but you can't usually get the whole team on board with Nix so version mismatches come up often and are often difficult to debug. You end up in a state where whoever wrote the initial setup scripts is authoritative about the dependencies, and you have to ask them what they have installed if you want things to work (it was easy for them, they just used whatever was lying around at the time).

These aren't directly deficiencies of k8s, once you see the light (which takes a long time) it's pretty easy to work with, but like so many other technologies, the devil is in the peripheral tooling and the culture. K8s doesn't (yet?) have a very nice boundary with other language ecosystems, it reminds me of Java in that way. The die hard k8s people often want to solve problems by bringing them more fully into the k8s way of seeing the world and I just don't think that is consistent enough with reality to be our everything.

I've cultivated a begrudging respect for it, but I still don't like it. If I break free and start my own company, I'll publish an operator so that my stuff can be installed into k8s, but I don't intend to make it primary in any way.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#32

Post by 8organicbits »

ssijak wrote: post #10

Is Kubernetes still hard in 2024?

I've seen projects run on Cloud Run because they fit that model. But if you don't quite fit the model, then you fight the abstraction in odd ways. K8s gives you control over more things, so the easy stuff feels hard but the harder things aren't as hard. With k8s operators you can stand up pretty complex things very quickly and robustly.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#33

Post by jen20 »

jhgg wrote: post #22
ysofunny wrote: post #8

once upon a time I set up an elastic search cluster in kubernetes after a lot of tweaking I made it so that the pods would be as big as the underlying hardware nodes. one pod one node. once that was working I realized that I was using the wrong tool for the job. the kubernetes tooling added nothing but complexity. needless to say I let it run like that having had wasted about a week getting it to work

On the other hand, at a certain scale (running hundreds of ES nodes across 80 or so ES clusters), Kubernetes actually does make a lot of sense. At work, we moved from hosting elastic search on bare VMs to kubernetes. By leveraging scheduler policies we are able to pack / over-provision ES node pods of different clusters onto the same Kubernetes nodes, allowing for far greater resource efficiency, while being able to…

A big part of the problem with Kubernetes is it doesn't make a ton of sense at small scale, and it just plain doesn't work at large scale.

Nomad is generally speaking a much more appropriate technology when you hit the point of needing such a system.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#34

Post by figassis »

renegade-otter wrote: post #5
gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

This kind of resilience is a form of art, and it's also kind of a full-time job. I would not advise trying this for a "side project at work". Generally we all agree that we move to the cloud and it's "fully managed". If it goes down - that's the price we pay. If you have to ask the question "what if the RDS goes down", then you are really in a different universe. That last guarantee of uptime requires a ton of work,…

My company also runs on a k8s cluster, but I agree with this fully and I keep my DB outside the cluster. For me, the order of preference is: RDS, Cloud Instance, Bare metal DB server, something else that is not kubernetes.

RDS will fail, but it likely will come back up without much action on my part. K8s will fail, you will spend untold human hours figuring out the k8s failure modes, before figuring out the database failure modes (which likely are quite straight forward). It's just a cost that is not worth it.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#35

Post by mikepurvis »

levkk wrote: post #17
gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

"K8s is not really for stateful systems"

As a relative novice in the space, I'm grateful to hear someone say this out loud. K8s seems perfect to me for quickly scaling transient stuff like pipeline workers, web servers, but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications, databases, and so on. And I've always felt kind of kind of guilty for not being on board to just mindlessly k8s-all-the-things.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#36

Post by marcosdumay »

jauntywundrkind wrote: post #21
gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

Is the complexity better or worse than alternatives? What are the alternatives? People use complexity as a boogieman to justify throwing together their own really wild chaotic & only so-so tested "simple" alternatives all the time . To me, this feels like a modern wonder. We have layers of responsibility. Many people operate Kubernetes clusters already for all kinds of reasons. It provides a powerful broad base. Now…

> What are the alternatives?

Installing the Postgres on the computer, or on a VM.

Really, just the fact that some people keep asking this question is enough to question everything else they say. The alternatives are obvious.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#37

Post by stanac »

iamgopal wrote: post #20
justanotheratom wrote: post #12

dumb question - where is the storage kept?

Me who have never used kubernetes, what if node crash ? Will I lost everything ?

No, attached storage is not part of the node (not directly). It's something like attaching external (host) directory to a docker container. Your can kill the node/pod and storage is not affected, later you can attach new pod to the same storage.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#38

Post by rad_gruchalski »

iamgopal wrote: post #20
justanotheratom wrote: post #12

dumb question - where is the storage kept?

Me who have never used kubernetes, what if node crash ? Will I lost everything ?

The answer is: it depends. It depends on if you use persistent volumes, and how well is your pv isolated from the failed node. If done right, no data loss.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#39

Post by doublerabbit »

bo0tzz wrote: post #25
justanotheratom wrote: post #12

dumb question - where is the storage kept?

Wherever you want, Kubernetes supports basically every storage backend you can imagine.

> Kubernetes supports basically every storage backend you can imagine.

Text files?

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#40

Post by marcosdumay »

jhgg wrote: post #22
ysofunny wrote: post #8

once upon a time I set up an elastic search cluster in kubernetes after a lot of tweaking I made it so that the pods would be as big as the underlying hardware nodes. one pod one node. once that was working I realized that I was using the wrong tool for the job. the kubernetes tooling added nothing but complexity. needless to say I let it run like that having had wasted about a week getting it to work

On the other hand, at a certain scale (running hundreds of ES nodes across 80 or so ES clusters), Kubernetes actually does make a lot of sense. At work, we moved from hosting elastic search on bare VMs to kubernetes. By leveraging scheduler policies we are able to pack / over-provision ES node pods of different clusters onto the same Kubernetes nodes, allowing for far greater resource efficiency, while being able to…

Did you keep adding and removing replicas into your cluster based on a scheduler policy? How often did you adjust the number of nodes (and how long did it take to make a node available)?

At the high-level you are describing your setup, it doesn't make sense. You'd spend way more resources managing any cluster than what you would gain from a normal-looking policy. I seem to be missing some important detail.

Post reply on HN