Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

github.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#41

Post by takeda »

tokenizerrr wrote: post #12
FooBarWidget wrote: post #11

I wonder how much things like this are needed going forward. From my (rudimentary) knowledge about Kubernetes and stateful sets, I think that Kubernetes is able to solve a lot of the issues surrounding failover, recovery of the old master, and partitioning; providing that we use Kubernetes in combination with networked storage that guarantees reliability. It appears that the way to setup PostgreSQL (or any replicated…

> providing that we use Kubernetes in combination with networked storage that guarantees reliability What kind of networked storage do people like with kubernetes? I've recently set up a small cluster not in any cloud, and persistent cross-node storage is a concern. There's quite a few options such as glusterfs, but I'd be curious to know if anyone here knows about the tradeoffs.

If you're in AWS then EFS is what is designed to do that, it is not cheap, but setting up a reliable solution is not easy either.

If your goal is file based storage (not block based), application can make api calls to get data and is mostly read, then S3 might also be an option.

Alternatively, roll-your-own on EC2 instances, which might not be as easy.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#42

Post by takeda »

cies wrote: post #5

This is very interesting. I thought it was best to keep yr db out of the k8s cluster. But this seems to make Postgres "cloud native". The main reason for keeping dbs out of k8s was that the storage (persistence) solutions in k8s were not up to the task yet. Now I wonder how is Patroni doing persistence? I cannot find anything on it in the Patroni docs. Maybe Patroni is so "self healing" that a proper storage solution…

Biggest issue, is not really k8s but docker, its drivers for storage are still quite young.

I personally wouldn't place a database in container for that reason. Essentially when you do so any durability guarantees that a database provides are going out of the window.

Things might seem to work fine, but when container get abruptly terminated in a wrong moment you might learn that all data is gone.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#43

Post by CyberDem0n »

bogomipz wrote: post #39
CyberDem0n wrote: post #36

Earlier quoted context omitted.

Compose guys are kind of happy with existing functionality of Governor and didn't really wanted to add something new. That was the main reason of making a fork. If you read a further comments to this news you can get familiar with the part of new functionality of Patroni comparing to Governor.

Thanks. Yeah I've been making my way through the video - "Elephants on Automatic: HA Clustered PostgreSQL with Helm" and its discussed at the 15:57 minute mark here if anyone else is interested: https://www.youtube.com/watch?v=CftcVhFMGSY Cheers.

You also can have a look on slides of my talk about at PGConf.US 2017: https://www.slideshare.net/AlexanderKukushkin1/patroni-ha-po...

Talk was also recorded but video is not yet published.

And one more thing, you can try to run a Live-Demo the same way as I did it during presentation: https://cyberdemn.blogspot.de/2017/04/patroni-ha-postgresql-...

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#44

Post by hintbits »

takeda wrote: post #42
cies wrote: post #5

This is very interesting. I thought it was best to keep yr db out of the k8s cluster. But this seems to make Postgres "cloud native". The main reason for keeping dbs out of k8s was that the storage (persistence) solutions in k8s were not up to the task yet. Now I wonder how is Patroni doing persistence? I cannot find anything on it in the Patroni docs. Maybe Patroni is so "self healing" that a proper storage solution…

Biggest issue, is not really k8s but docker, its drivers for storage are still quite young. I personally wouldn't place a database in container for that reason. Essentially when you do so any durability guarantees that a database provides are going out of the window. Things might seem to work fine, but when container get abruptly terminated in a wrong moment you might learn that all data is gone.

Why would the data disappear, given that it's stored on an external volume (i.e. EBS or Google Cloud Storage or just a directory on your laptop), not in the container FS?

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#45

Post by geoka9 »

CyberDem0n wrote: post #40
geoka9 wrote: post #38

What's zalando/patroni's approach to performance scaling/load balancing? For example, pgpool2 can distribute SELECTs among multiple slaves which makes it useful for heavy analytics loads. Can you guys do that, or you only handle HA?

Patroni has REST API providing a health-checks for load-balancers. patroni:8008/replica will respond with http status code 200 only if node is running as replica. In the HAProxy config file you just need to list all Patroni nodes, specify a health-check and it will do a load-balancing for you. The same approach works for example with AWS ELB.

But writes need to be directed to the master, right? Or Patroni is smart enough to do it by itself?

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#46

Post by bogomipz »

CyberDem0n wrote: post #43
bogomipz wrote: post #39

Earlier quoted context omitted.

Thanks. Yeah I've been making my way through the video - "Elephants on Automatic: HA Clustered PostgreSQL with Helm" and its discussed at the 15:57 minute mark here if anyone else is interested: https://www.youtube.com/watch?v=CftcVhFMGSY Cheers.

You also can have a look on slides of my talk about at PGConf.US 2017: https://www.slideshare.net/AlexanderKukushkin1/patroni-ha-po... Talk was also recorded but video is not yet published. And one more thing, you can try to run a Live-Demo the same way as I did it during presentation: https://cyberdemn.blogspot.de/2017/04/patroni-ha-postgresql-...

Oh neat, thanks!

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#47

Post by CyberDem0n »

geoka9 wrote: post #45
CyberDem0n wrote: post #40

Earlier quoted context omitted.

Patroni has REST API providing a health-checks for load-balancers. patroni:8008/replica will respond with http status code 200 only if node is running as replica. In the HAProxy config file you just need to list all Patroni nodes, specify a health-check and it will do a load-balancing for you. The same approach works for example with AWS ELB.

But writes need to be directed to the master, right? Or Patroni is smart enough to do it by itself?

Correct. Writes need to be directed to master. May be not only writes but some reads as well. Only application can know which statement can be executed on replicas and which must be executed on master.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#48

Post by smarterclayton »

foxylion wrote: post #15
FooBarWidget wrote: post #11

I wonder how much things like this are needed going forward. From my (rudimentary) knowledge about Kubernetes and stateful sets, I think that Kubernetes is able to solve a lot of the issues surrounding failover, recovery of the old master, and partitioning; providing that we use Kubernetes in combination with networked storage that guarantees reliability. It appears that the way to setup PostgreSQL (or any replicated…

I think the largest disadvantage is, that you won't get real high availability. Promoting a slave to a master is done in a second. Restarting the master can take some time (especially when it crashed previously). There is also the problem of losing the storage volume of your master node (ebs block storage do neither have 100% availability, nor 100% durability). In this case you can't recover your master.

This is correct. The time to recreate a stateful set pod is:

1. Time to detect node failure

2. Time for cloud provider to indicate node down

3. Time to create new pod and wait for startup

Today, 1/2 can be from 20-30s to infinite. Kubernetes does not drain a dead node unless it receives some external signal that the node is truly dead.

Future work involves adding such failure detection with a fencer (that can ensure the node is partitioned, at which point it's safe to drain it). Just like normal HA, Kubernetes needs to know the node is actually down, vs just partitioned.

I would recommend designing your stateful sets so that you can move master from index 0 to index N in the event of a missed heartbeat, and make sure your clients are accessing the master via some proxy (whether kubeproxy or another). If the node dies under partition, but clients are accessing via the service IP, starting deletion of the pod will update the service for any non-partitioned nodes.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#49

Post by smarterclayton »

takeda wrote: post #42
cies wrote: post #5

This is very interesting. I thought it was best to keep yr db out of the k8s cluster. But this seems to make Postgres "cloud native". The main reason for keeping dbs out of k8s was that the storage (persistence) solutions in k8s were not up to the task yet. Now I wonder how is Patroni doing persistence? I cannot find anything on it in the Patroni docs. Maybe Patroni is so "self healing" that a proper storage solution…

Biggest issue, is not really k8s but docker, its drivers for storage are still quite young. I personally wouldn't place a database in container for that reason. Essentially when you do so any durability guarantees that a database provides are going out of the window. Things might seem to work fine, but when container get abruptly terminated in a wrong moment you might learn that all data is gone.

Kubernetes does not use Docker's storage drivers. There have been bugs with attach/detach of cloud block storage though that has impacted applications a few times over Kubernetes lifecycle, although all have been fixed fairly quickly.

Re: Patroni: A Template for PostgreSQL HA with ZooKeeper, Etcd, or Consul

#50

Post by takeda »

hintbits wrote: post #44
takeda wrote: post #42

Earlier quoted context omitted.

Biggest issue, is not really k8s but docker, its drivers for storage are still quite young. I personally wouldn't place a database in container for that reason. Essentially when you do so any durability guarantees that a database provides are going out of the window. Things might seem to work fine, but when container get abruptly terminated in a wrong moment you might learn that all data is gone.

Why would the data disappear, given that it's stored on an external volume (i.e. EBS or Google Cloud Storage or just a directory on your laptop), not in the container FS?

I simplified, because in the end that would be the equivalent.

The danger is that some bugs in a filesystem, manifests themselves only when there is abrupt termination, otherwise things appear normal. Stuff like data is being written in incorrect order, or data is synced to disk at wrong time, or not at all.

On top of that user also expects performance, so some shortcuts can be taken that compromises the above (especially if the criteria is that docker is intended for stateless applications).

When there bugs like that and things are abruptly terminated, the data on the disk can be heavily mangled to a point that can't be recovered so essentially it would be lost.

Post reply on HN