Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-hosting a high-availability Postgres cluster on Kubernetes

ryan-schachte.com

Post reply on HN
Print view View original

81–90 of 92 posts

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#81

Post by debarshri »

SlightlyLeftPad wrote: post #62
debarshri wrote: post #45

Earlier quoted context omitted.

Problem is that RDS comes at a price. It is purely about operation cost. When you have 1500+ databases these cost add up. At that point, this kind of techniques are required to self host the databases. Price per DB with HPA na VPA is way lower than what you would pay for managed databases as well as you can hire a full time devops+dbadmin and still be cheaper.

This is a perfect example of a cost savings opportunity that offers no benefit and is really just a foot gun when it comes to operational complexity. The cloud cost savings almost never justify the operational costs here. There’s typically a very long list of other cost optimizations to be made before this would ever be on the table for me. If you’re talking about 1500+ databases and think a single full time devops+d…

I am not implying hiring single full time dbadmin, you can hire a team of them and still be cheaper that what AWS would cost at that scale.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#82

Post by seanhunter »

p_l wrote: post #79
seanhunter wrote: post #46

Earlier quoted context omitted.

Exactly. The way people successfully did HA and scalability on databases for more than 20 years before Kubernetes existed.

With a bunch of hand crafted, hairy code to link the necessary features like fail over, backups, PITR, clustering. I know, I did all of that. Which is why I punt that effort to a k8s operator these days, because at the end of the day it does everything I did manually plus makes it easier for me to spin a new database that has WAL shipping and backups to different location.

If you're punting to an operator why k8s? There are lots of managed database providers and if they're managing it, then k8s (or not) is an implementation detail for them to worry about.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#83

Post by cjaro »

danielvaughn wrote: post #23
ssijak wrote: post #10

Is Kubernetes still hard in 2024?

Granted I'm new to devops, only been on a platform team for about 9 months now, but I still feel incredibly dumb every time I try to work with it. That being said, we're also layering a bunch of stuff on top - helm, nginx, GKE, terraform, as well as a mountain of other things, and then to top it off we have a bunch of shell scripts doing random things to help tie it all together. Normally I can pick things up pretty…

I appreciate your shared experience as I have a rather similar one. I've been on a platform SRE team for about eight months myself, with seven years of SWE experience before that, and feel as though I'm just able keep my head above water. It's one thing to learn about k8s, the cloud, terraform, etc, then quite another to pile it all together, particularly since it all becomes heavily customized. It's a different job than code-writing software engineering, that's for sure. To me, it feels less like there's a 'stack' so much as there's a word cloud of DevOps buzzwords to start throwing at problems. Even when directed by architects & principals, it overwhelms.

I resonate with feeling incredibly dumb whenever I pick up a new ticket from our backlog. It feels like gaining deep knowledge of these systems will be a nearly insurmountable challenge. It's been eight months, and while I know far, far more than I did on day one, I feel that every day is a day one of sorts.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#84

Post by dilyevsky »

seanhunter wrote: post #82
p_l wrote: post #79

Earlier quoted context omitted.

With a bunch of hand crafted, hairy code to link the necessary features like fail over, backups, PITR, clustering. I know, I did all of that. Which is why I punt that effort to a k8s operator these days, because at the end of the day it does everything I did manually plus makes it easier for me to spin a new database that has WAL shipping and backups to different location.

If you're punting to an operator why k8s? There are lots of managed database providers and if they're managing it, then k8s (or not) is an implementation detail for them to worry about.

Lots of various reasons. Performance, cost, compliance, and reliability are the ones I've personally seen.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#85

Post by jauntywundrkind »

seanhunter wrote: post #82
p_l wrote: post #79

Earlier quoted context omitted.

With a bunch of hand crafted, hairy code to link the necessary features like fail over, backups, PITR, clustering. I know, I did all of that. Which is why I punt that effort to a k8s operator these days, because at the end of the day it does everything I did manually plus makes it easier for me to spin a new database that has WAL shipping and backups to different location.

If you're punting to an operator why k8s? There are lots of managed database providers and if they're managing it, then k8s (or not) is an implementation detail for them to worry about.

These are each specific DB-as-a-services. Instead of sinking my time into learning something universal & competent & capable at many things (Kubernetes) now I'm investing in learning & using a niche service, that I won't have fine control over & whose limits might only be clear down the road. That's a risk.

Sure the dbaas might have superb autoscing and reliability. It may or may not have good OpenTofu/terraform providers available, or other custom niche deployment tools one would need to pick up & adapt.

Same questions as roughing it hacking together pg yourself, except now you have no power. Is observability going to be up to snuff? When performance problems come, how are you going to feel when you discover that analyst techniques a and b work fine but c and d can't be done because you are on managed service that doesn't actually give you full access to the db?

Costs can also add up, and are hard to predict. If you have a managed service, you'll be using a lot more network, which often has some cost (also some latency too). This isn't for everyone, but Kube makes it easy to start with anything (ex: RPi) and scale out. Works with hosted hardware with some local SSD, or I can get a 16-core 7950X and a bunch of gen 5 nvme drives and a fraction of a terabyte of ram for I think it's so so ideal to have a reliably capable universal autonomic computing later underfoot. Most of the backend has had special specific answers to each question, each problem. We have been disjointed. Kubernetes provides a platform applicable to a vast range of workloads, where yes you need to tune and build different clusters for different workloads, but where the skills transfer much more readily. Everyone in Kubernetes knows how to ship resources definitions/manifests, and that base knowledge is all you need to learn to start consuming services - any services - on Kubernetes. Skill transfer is immediate. Having universal patterns (the API server), backed by autonomic operators making & keeping these resources real: it's so much better than the thousand different paths road of technology we've been on until now.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#86

Post by jhgg »

marcosdumay wrote: post #40
jhgg wrote: post #22

Earlier quoted context omitted.

On the other hand, at a certain scale (running hundreds of ES nodes across 80 or so ES clusters), Kubernetes actually does make a lot of sense. At work, we moved from hosting elastic search on bare VMs to kubernetes. By leveraging scheduler policies we are able to pack / over-provision ES node pods of different clusters onto the same Kubernetes nodes, allowing for far greater resource efficiency, while being able to…

Did you keep adding and removing replicas into your cluster based on a scheduler policy? How often did you adjust the number of nodes (and how long did it take to make a node available)? At the high-level you are describing your setup, it doesn't make sense. You'd spend way more resources managing any cluster than what you would gain from a normal-looking policy. I seem to be missing some important detail.

You must be. It is very easy. We just let kubernetes scheduler take care of everything. We use anti affinities to disallow scheduling nodes from the same ES cluster on the same K8S node. We use the ES operator which will apply the appropriate PDBs to ensure that operations done to the k8s cluster as a whole doesn't end up turning off too many things, and threatening availability.

Things literally have just worked.

We are about to do a scale up operation of about 6 nodes each for 16 clusters (total of 96 ES data nodes being added.) This was editing a variable in a config file, and pushing that config to kubernetes. The operation will probably take a day to complete as we throttle the speed at which data transfers to new nodes to not adversely affect latency or ingest rate.

The amount of "human time" to kick off this operation and begin the scale up is measured in minutes, then it's just letting it do its thing.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#87

Post by jhgg »

jen20 wrote: post #33
jhgg wrote: post #22

Earlier quoted context omitted.

On the other hand, at a certain scale (running hundreds of ES nodes across 80 or so ES clusters), Kubernetes actually does make a lot of sense. At work, we moved from hosting elastic search on bare VMs to kubernetes. By leveraging scheduler policies we are able to pack / over-provision ES node pods of different clusters onto the same Kubernetes nodes, allowing for far greater resource efficiency, while being able to…

A big part of the problem with Kubernetes is it doesn't make a ton of sense at small scale, and it just plain doesn't work at large scale. Nomad is generally speaking a much more appropriate technology when you hit the point of needing such a system.

I would consider our scale pretty large here, and it works just fine.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#88

Post by doublerabbit »

dilyevsky wrote: post #74
doublerabbit wrote: post #39

Earlier quoted context omitted.

> Kubernetes supports basically every storage backend you can imagine. Text files?

If you can figure out how to create mount point out of it and provide your own csi driver then yes

So that's a no then.

Creating != Supporting

Anything can be supported by creating your own driver.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#89

Post by gnufied »

levkk wrote: post #17
gchamonlive wrote: post #4

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

I am the author of k8s resizing feature and its been GAed for awhile and feedback we have got so far has been good. If anything running inside k8s makes it relatively easy to support resizing. You just need to specify new size for PVC and it will both perform resizing on the cloudprovider and of the file system (if needed).

Modifying IOPS and other volume attributes is something less frequently needed but we just released alpha support for that too, if you must need it.

We have also added support for reporting volume usage in CSI specs, which I know some operators use to automatically resize volumes when certain threshold is reached (I however do not recommend using ephemeral metrics for automating something like this). But point is - you can actually define CRDs that persist volume usage and have it used by an higher level operator.

Another thing is - k8s makes it relatively easy to take snapshots which can be automated too and that should give someone additional peace of mind if something goes haywire.

Obviously I am biased and I know there are some lingering issues that require manual intervention when using stateful workloads (such as when a node crashes), but k8s should be just as good for running stateful workloads IMO.

Another thing is - k8s volumes are nothing but bind mounts from host namespace into container's namespace and hence there should be no performance penalty of using them.

Re: Self-hosting a high-availability Postgres cluster on Kubernetes

#90

Post by gnufied »

mikepurvis wrote: post #35
levkk wrote: post #17

Earlier quoted context omitted.

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

"K8s is not really for stateful systems" As a relative novice in the space, I'm grateful to hear someone say this out loud. K8s seems perfect to me for quickly scaling transient stuff like pipeline workers, web servers, but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications, dat…

>but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications,

You can do that too with k8s with APIs which support more than just one backend.

Post reply on HN