Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Postgres in Kubernetes [pdf]

static.sched.com

Post reply on HN
Print view View original

51–60 of 101 posts

Re: Running Postgres in Kubernetes [pdf]

#51

Post by johncolanduoni »

GordonS wrote: post #48
johncolanduoni wrote: post #43

Earlier quoted context omitted.

Kubernetes can’t change any database’s HA or durability features; there’s no magic k8s can apply to make a database that does e.g. asynchronous replication have the properties of one that does synchronous replication. So you’ll never gain any properties your underlying database is incapable of providing. However, if I had to run Postgres as part of something I deployed on k8s AND for some reason couldn’t use my cloud…

> AND for some reason couldn’t use my cloud provider’s built in solution (AWS RDS, Cloud SQL, etc.) Money. > Pay someone else to handle that part Aside from the money (managed Postgres is expensive ), I'd actually like to understand what good, high-availability Postgres solutions look like.

Totally valid if you’re reaching the larger instance sizes. However I’d caution you to not underestimate what running a good HA Postgres setup costs in engineering/operations time (particularly if you’re not familiar with running one). Be ready to get a clearheaded TCO that you won’t be happy with.

In general, you need some sort of separate strictly serializable store run in an HA configuration to manage the state changes, and whatever is managing the state changes needs to be run in multiple instances across your fault domains as well. Others have mentioned Patroni; I’ve used it before (though with etcd, not k8s) and been quite happy with it. Be aware that (as it cautions you in the README) it’s not a total point and shoot tool, you do need to read through it’s caveats and understand the underlying Postgres replication features.

The documentation is pretty good, if you want to get an idea of what the logic looks like they have a nice state diagram: https://github.com/zalando/patroni/blob/master/docs/ha_loop_...

Re: Running Postgres in Kubernetes [pdf]

#52

Post by random3 »

georgebarnett wrote: post #33
random3 wrote: post #26

Earlier quoted context omitted.

GCP volumes are over network already. You can deploy stateful workloads using StatefulSets. We've run an HBase workloads for development purposes (about 20-30x cheaper than BigTable) and it worked great (no issues for over 12 months). While Postgres is hardly a distributed database, there may be some advantages to ensure availability and perhaps even more in replicated setup.

Why are you comparing HBase with Postgres? They are very different technologies with completely different architectural constraints?

They both require persistence semantics. I forgot to mention, I was referring to a single node, no-HDFS setup for HBase, solely relying on Kubernetes StatefulSets for data availability, in the simplified persistene sense, not much different than a single Posgres server.

Re: Running Postgres in Kubernetes [pdf]

#53

Post by lstamour »

ci5er wrote: post #50
jonfw wrote: post #34

Earlier quoted context omitted.

The nice thing about running a DB inside a cluster is running your entire application, end to end, through one unified declarative model. It's really really easy to spin up a brand new dev or staging environment. generally though in production, you're not going to be taking down DBs on purpose. If it's not supposed to be ephemeral, it doesn't fit the model

My biases are probably 5+ years old, but it used to be that running PostGres on anything other than bare metal (for example, running it in Docker) was fraught with all sorts of data/file/io sync issues that you might not be able to recover from. So, I just got used to running the databases on metal (graph, postgres, columnar) with whatever reliability schemes myself and leaving docker and k8s outside of that. Has tha…

I might have lost count but I think they reimplemented the file storage twice in local Docker installs and twice in Kubernetes... it’s at a point now where if you trust cloud network storage performance guarantees, you can trust Postgres with it. Kubernetes and Docker don’t change how the bits are saved, but if you insist on data locality to a node, you lose the resilience described here for moving to another node.

Here’s a different point to think about: is your use of Postgres resilient to network failures, communication errors or one-off issues? Sometimes you have to design for this at the application layer and assume things will go wrong some of the time...

As with anything, it could vary with your particular workload... but if I knew my very-stable-yet-cloud-hosted copy of Postgres wasn’t configured with high availability, well, you might have local performance and no update lag but you also have a lot of risk of downtime and data loss if it goes down or gets corrupted. The advantage to cloud storage is not having to read in as many WAL logs, and just reconnect the old disk before the instance went down, initialize as if PG had just crashed, and keep going... even regular disks have failures after all...

Re: Running Postgres in Kubernetes [pdf]

#54

Post by random3 »

GordonS wrote: post #35
random3 wrote: post #27

Earlier quoted context omitted.

The main advantage with Kubernetes (especially in low ops environments like GKE) is not scalability, but availability and ease of development (spinning things up and down is super-easy). The learning curve to stand something up is not very high and pays of over time compared to SSH-ing into VMs.

I'm very comfortable with containers (less so specifically with k8s), but generally for stateless or stateless'ish services. What are the advantages of k8s specifically for a database?

High availability, deployment consistency and, if needed, ability to scale-out on demand to name a few. You can can have an all-inclusive environment for sandboxing, development or production use-cases. Easy to spin-up and tear down.

A good alternative is to use a database service and not have to worry about it at all. However if you do have to operate one, orchestration systems give you the tools to handle it properly WRT your availability and performance SLA/SLOs.

Re: Running Postgres in Kubernetes [pdf]

#55

Post by georgebarnett »

random3 wrote: post #52
georgebarnett wrote: post #33

Earlier quoted context omitted.

Why are you comparing HBase with Postgres? They are very different technologies with completely different architectural constraints?

They both require persistence semantics. I forgot to mention, I was referring to a single node, no-HDFS setup for HBase, solely relying on Kubernetes StatefulSets for data availability, in the simplified persistene sense, not much different than a single Posgres server.

Under those conditions I'd agree theres likely a high overlap of similarity. I have a use case on my home playpen that would likely be served well by that. Thanks for describing your approach.

Re: Running Postgres in Kubernetes [pdf]

#56

Post by gavinray »

ci5er wrote: post #50
jonfw wrote: post #34

Earlier quoted context omitted.

The nice thing about running a DB inside a cluster is running your entire application, end to end, through one unified declarative model. It's really really easy to spin up a brand new dev or staging environment. generally though in production, you're not going to be taking down DBs on purpose. If it's not supposed to be ephemeral, it doesn't fit the model

My biases are probably 5+ years old, but it used to be that running PostGres on anything other than bare metal (for example, running it in Docker) was fraught with all sorts of data/file/io sync issues that you might not be able to recover from. So, I just got used to running the databases on metal (graph, postgres, columnar) with whatever reliability schemes myself and leaving docker and k8s outside of that. Has tha…

My "run-of-the-mill" setup for local toy apps is Postgres + [some API service] in Docker Compose, and for non-trivial apps (IE more than 3-4 services) usually Postgres in k8s.

I've never had a problem with Postgres either in Docker or in k8s. Docker Compose local volumes, and k8s persistent volume claims work really well. But I'm no veteran at this so I can only speak for what little time I've used them.

The whole reason I do this is because it lets you put your entire stack in one config, and then spin up local dev environment or deploy to remote with zero changes. And that's really magical.

In production I don't use an in-cluster Postgres, and it's a bit of a pain in the ass to do so. I would rather use an in-cluster service, but the arguments you hear about being responsible in the event of a failure, and the vendor assuming the work for HA and such seems hard to refute.

Probably you could run Postgres in production k8s and be fine though. If I knew what I was doing I likely wouldn't be against it.

Re: Running Postgres in Kubernetes [pdf]

#57

Post by ci5er »

lstamour wrote: post #53
ci5er wrote: post #50

Earlier quoted context omitted.

My biases are probably 5+ years old, but it used to be that running PostGres on anything other than bare metal (for example, running it in Docker) was fraught with all sorts of data/file/io sync issues that you might not be able to recover from. So, I just got used to running the databases on metal (graph, postgres, columnar) with whatever reliability schemes myself and leaving docker and k8s outside of that. Has tha…

I might have lost count but I think they reimplemented the file storage twice in local Docker installs and twice in Kubernetes... it’s at a point now where if you trust cloud network storage performance guarantees, you can trust Postgres with it. Kubernetes and Docker don’t change how the bits are saved, but if you insist on data locality to a node, you lose the resilience described here for moving to another node. H…

That's great to know. Thank you. The stuff in your 3rd paragraph is something I have always tried to do, but it largely depended on the largely reliable single node instances. I tend to (currently) handle single disk failures with master-slave configs locally, and it has worked at my volumes to date, but I am trying to learn how to grow without getting increasingly (too) complicated or even more brittle.

Re: Running Postgres in Kubernetes [pdf]

#58

Post by ci5er »

gavinray wrote: post #56
ci5er wrote: post #50

Earlier quoted context omitted.

My biases are probably 5+ years old, but it used to be that running PostGres on anything other than bare metal (for example, running it in Docker) was fraught with all sorts of data/file/io sync issues that you might not be able to recover from. So, I just got used to running the databases on metal (graph, postgres, columnar) with whatever reliability schemes myself and leaving docker and k8s outside of that. Has tha…

My "run-of-the-mill" setup for local toy apps is Postgres + [some API service] in Docker Compose, and for non-trivial apps (IE more than 3-4 services) usually Postgres in k8s. I've never had a problem with Postgres either in Docker or in k8s. Docker Compose local volumes, and k8s persistent volume claims work really well. But I'm no veteran at this so I can only speak for what little time I've used them. The whole re…

Ha! That's probably the problem - I don't know what I am doing.

Why oh why did I ever leave silicone...

Re: Running Postgres in Kubernetes [pdf]

#59

Post by qeternity »

penagwin wrote: post #38
GordonS wrote: post #25

So, a bit OT, but I'm looking for some advice on building a Postgres cluster, and I'm pretty sure k8s is going to add a lot of complexity with no benefit. I'm a Postgres fan, and use it a lot, but I've never actually used it in a clustered setup. What I'm looking at clustering for is not really for scalability (still at the stage where we can scale vertically), but for high availability and backup - if one node is do…

Compared to many databases, postgres HA is a mess. It has builtin streaming, but no fail over of any kind, all of that has to be managed by another application. We've had the best luck with patron, but even then you'll find the documentation confusing, have weird issues, etc. You'll need to setup etcd/Consul to use it. That's right you need a second database cluster to setup your database cluster.... Great... I have…

Very true. My sentiments exactly as Spilo/Patroni users. One benefit to k8s is you can use it as the DCS for Patroni

Re: Running Postgres in Kubernetes [pdf]

#60

Post by lazyant »

ci5er wrote: post #50
jonfw wrote: post #34

Earlier quoted context omitted.

The nice thing about running a DB inside a cluster is running your entire application, end to end, through one unified declarative model. It's really really easy to spin up a brand new dev or staging environment. generally though in production, you're not going to be taking down DBs on purpose. If it's not supposed to be ephemeral, it doesn't fit the model

My biases are probably 5+ years old, but it used to be that running PostGres on anything other than bare metal (for example, running it in Docker) was fraught with all sorts of data/file/io sync issues that you might not be able to recover from. So, I just got used to running the databases on metal (graph, postgres, columnar) with whatever reliability schemes myself and leaving docker and k8s outside of that. Has tha…

There's only network overhead when running Postgres in Docker container (and obv data is mounted, not ephemeral), but it's a performance hit. This in itself can be enough reason not to run pg in Docker/k8s in production.
Post reply on HN