Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

github.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#21

Post by abkfenris »

rsanders wrote: post #10
cies wrote: post #5

This is very interesting. I thought it was best to keep yr db out of the k8s cluster. But this seems to make Postgres "cloud native". The main reason for keeping dbs out of k8s was that the storage (persistence) solutions in k8s were not up to the task yet. Now I wonder how is Patroni doing persistence? I cannot find anything on it in the Patroni docs. Maybe Patroni is so "self healing" that a proper storage solution…

There are many solutions for network block device style persistent storage in K8S now. In the cloud, you have solutions like Amazon's EBS and GCE's persistent disks. On VMware you have local VMDKs. On your own equipment, there's support for iSCSI, Ceph RBD, OpenStack Cinder, Portworx, ScaleIO, Quobyte, and just about anything else if you write a few scripts to format and mount volumes on the host (FlexVolume). Except…

It's a bit scary when you're using K8S ephemeral storage and depending on a basebackup from another pod, but you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest). That way any time you lose a pod, you get a test of your backup system on restore.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#22

Post by eicnix »

abkfenris wrote: post #21
rsanders wrote: post #10

Earlier quoted context omitted.

There are many solutions for network block device style persistent storage in K8S now. In the cloud, you have solutions like Amazon's EBS and GCE's persistent disks. On VMware you have local VMDKs. On your own equipment, there's support for iSCSI, Ceph RBD, OpenStack Cinder, Portworx, ScaleIO, Quobyte, and just about anything else if you write a few scripts to format and mount volumes on the host (FlexVolume). Except…

It's a bit scary when you're using K8S ephemeral storage and depending on a basebackup from another pod, but you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest). That way any time you lose a pod, you get a test of your backup system on restore.

There is currently some work being done for improving local storage usage in k8s. https://github.com/kubernetes/community/pull/306

Additionally k8s can be configured to restart pods on the same node so you can use the local node storage for persistence and rely on your DB for replication.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#23

Post by jan_berlin »

vishesh92 wrote: post #6
cies wrote: post #5

This is very interesting. I thought it was best to keep yr db out of the k8s cluster. But this seems to make Postgres "cloud native". The main reason for keeping dbs out of k8s was that the storage (persistence) solutions in k8s were not up to the task yet. Now I wonder how is Patroni doing persistence? I cannot find anything on it in the Patroni docs. Maybe Patroni is so "self healing" that a proper storage solution…

I am not really sure Patroni has anything to do with the persistence of data. It just uses etcd, zookeper or consul to elect a master in case of a failover and uses their key value store to save the information about the current master. Its upto you whether you keep your database on a container or not. And how the data is managed by the container. This talk by Josh Berkus explains how Patroni works pretty well. https…

The great thing today is there are lots of options with Patroni, it is very much up to you and your requirements what kind of storage you select and how you want to recover from node and storage failure.

Patroni with Spilo e.g. relies on EBS or local discs and can also ship WAL to S3. On Google it uses their equivalent solutions.

On Kubernetes earlier posts are correct that you may rely on Kubernetes bringing your pod back with the same underlying volume thus providing some kind of availability, but for others this is not good enough, e.g. remember EBS is single AZ only. Patroni can help here, running and orchestrating either slaves or giving you automated recovery from S3.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#24

Post by ZalandoTech »

Zalando open source evangelist here. Some of my colleagues are on this thread, taking your questions. Meanwhile, wanted to plug a few other Postgres/K8s projects we're working on:

-- Spilo: HA PostgreSQL cluster using Docker (https://github.com/zalando/spilo)

-- kube-ingress-aws-controller: Configures AWS Load Balancers according to Kubernetes Ingress resources (https://github.com/zalando-incubator/kube-ingress-aws-contro...)

-- external-dns: a Kubernetes Incubator collaboration. Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services. (https://github.com/kubernetes-incubator/external-dns)

We also have several older Postgres-related tools listed here: http://zalando.github.io/.

Users and contributors very welcome; just drop a line in our projects' Issues Trackers.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#25

Post by noway421 »

tokenizerrr wrote: post #12
FooBarWidget wrote: post #11

I wonder how much things like this are needed going forward. From my (rudimentary) knowledge about Kubernetes and stateful sets, I think that Kubernetes is able to solve a lot of the issues surrounding failover, recovery of the old master, and partitioning; providing that we use Kubernetes in combination with networked storage that guarantees reliability. It appears that the way to setup PostgreSQL (or any replicated…

> providing that we use Kubernetes in combination with networked storage that guarantees reliability What kind of networked storage do people like with kubernetes? I've recently set up a small cluster not in any cloud, and persistent cross-node storage is a concern. There's quite a few options such as glusterfs, but I'd be curious to know if anyone here knows about the tradeoffs.

The set up op describing is more suited for clouds I guess, where networked storage is already a ready to use feature like aws ebs block storage.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#27

Post by icebraining »

abkfenris wrote: post #21
rsanders wrote: post #10

Earlier quoted context omitted.

There are many solutions for network block device style persistent storage in K8S now. In the cloud, you have solutions like Amazon's EBS and GCE's persistent disks. On VMware you have local VMDKs. On your own equipment, there's support for iSCSI, Ceph RBD, OpenStack Cinder, Portworx, ScaleIO, Quobyte, and just about anything else if you write a few scripts to format and mount volumes on the host (FlexVolume). Except…

It's a bit scary when you're using K8S ephemeral storage and depending on a basebackup from another pod, but you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest). That way any time you lose a pod, you get a test of your backup system on restore.

you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest)

That sounds interesting, has anyone documented that kind of setup?

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#28

Post by jchw »

ZalandoTech wrote: post #24

Zalando open source evangelist here. Some of my colleagues are on this thread, taking your questions. Meanwhile, wanted to plug a few other Postgres/K8s projects we're working on: -- Spilo: HA PostgreSQL cluster using Docker ( https://github.com/zalando/spilo ) -- kube-ingress-aws-controller: Configures AWS Load Balancers according to Kubernetes Ingress resources ( https://github.com/zalando-incubator/kube-ingress-aw…

Looking forward to external-dns being merged. DNS records are probably the last thing in our Kubernetes cluster that isn't automated.

I'm still not sold on using ALBs for Ingress resources. ALB is clearly superior to ELB when using them for the same things, but it seems like ALB costs will soar when using ALBs as Ingress controllers since they increased the rule limit and subsequently started charging on them. Still, the alternative of having four layers for every route seems nearly as undesirable. I really don't want ALB -> nginx -> kube-proxy -> container.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#29

Post by abkfenris »

icebraining wrote: post #27
abkfenris wrote: post #21

Earlier quoted context omitted.

It's a bit scary when you're using K8S ephemeral storage and depending on a basebackup from another pod, but you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest). That way any time you lose a pod, you get a test of your backup system on restore.

you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest) That sounds interesting, has anyone documented that kind of setup?

I'm working on one. I am waiting for a PR to land and a release to get generated for a bug that I found.

If the master pod in a StatefulSet is deleted, and kubernetes sets up a new one too fast, it currently keeps the master lock and failover never happens (it also tries to bootstrap from itself).

Once those land I'll work on a writeup of that setup

I've got a few Postgres instances running as single node Deployments and StatefulSets right now that backup and restore using pghoard for simple services that I've documented here http://alexkerney.com/2016/10/pghoard-kubernetes.html I do need to add a note about using livenessprobes to make sure the restore completed successfully. Sometimes these pods are cycled every day as half my K8S cluster is preemptable.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#30

Post by hjacobs »

jchw wrote: post #28
ZalandoTech wrote: post #24

Zalando open source evangelist here. Some of my colleagues are on this thread, taking your questions. Meanwhile, wanted to plug a few other Postgres/K8s projects we're working on: -- Spilo: HA PostgreSQL cluster using Docker ( https://github.com/zalando/spilo ) -- kube-ingress-aws-controller: Configures AWS Load Balancers according to Kubernetes Ingress resources ( https://github.com/zalando-incubator/kube-ingress-aw…

Looking forward to external-dns being merged. DNS records are probably the last thing in our Kubernetes cluster that isn't automated. I'm still not sold on using ALBs for Ingress resources. ALB is clearly superior to ELB when using them for the same things, but it seems like ALB costs will soar when using ALBs as Ingress controllers since they increased the rule limit and subsequently started charging on them. Still,…

This is a bit off topic, but the mentioned Ingress controller (https://github.com/zalando-incubator/kube-ingress-aws-contro...) is not using the ALB rules, it just provisions ALBs with the right SSL certificate and points to some HTTP proxy doing the actual host/path routing (e.g. Skipper). See http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guid...
Post reply on HN