Introducing WAL-G: Faster Disaster Recovery for Postgres

Post by paulsutter » Fri, Aug 18, 2017, 10:43 PM UTC

ozgune wrote: post #33

drob wrote: post #20

This is great. Can't wait to be using it. We've been using WAL-E for years and this looks like a big improvement. The steady, high throughput is a big deal – our prod base backups take 36 hours to restore, so if the recovery speed improvements are as advertised, that's a big win. In the kind of situation in which we'd be using these, the difference between 9 hours and 36 hours is major. Also, the quality of life impr…

Hey Dan, nice to hear from you! > our prod base backups take 36 hours to restore, so if the recovery speed improvements are as advertised, that's a big win. Yes, if you attach 16TB of storage to each instance, your back-up restores may take a while. :))

Post by bpineau » Fri, Aug 18, 2017, 11:11 PM UTC

kafkes wrote: post #16

sehrope wrote: post #7

Earlier quoted context omitted.

What's the min version of the Postgres server this can be used with?

WAL-G uses non-exclusive backups so at least 9.6

Post by fdr » Fri, Aug 18, 2017, 11:48 PM UTC

upbeatlinux wrote: post #42

fdr wrote: post #36

Earlier quoted context omitted.

Do you want GPG based or some other client side encryption, or S3's encryption support? The latter could probably just be turned on. The former is a feature requiring code.

Ideally the presence of the `WALE_GPG_KEY_ID` env var should enable encrypted backups https://github.com/wal-e/wal-e#encryption . Put differently to be a "successor" it needs to be a drop in replacement ;)

Post by gardnr » Sat, Aug 19, 2017, 12:24 AM UTC

humanfromearth wrote: post #28

drob wrote: post #20

This is great. Can't wait to be using it. We've been using WAL-E for years and this looks like a big improvement. The steady, high throughput is a big deal – our prod base backups take 36 hours to restore, so if the recovery speed improvements are as advertised, that's a big win. In the kind of situation in which we'd be using these, the difference between 9 hours and 36 hours is major. Also, the quality of life impr…

Same here, perf is obviously great, but no more crazy dependencies is just great! So much time I wasted trying to make it fit into a docker container.

Post by daurnimator » Sat, Aug 19, 2017, 3:29 AM UTC

ngrilly wrote: post #44

andruby wrote: post #40

Earlier quoted context omitted.

It does continuous backup like WAL-E. Both back up PG's WAL files (Write Ahead Log) and allow restoring your database state as it was at a specific time or after a specific transaction committed. This is known as point-in-time recovery (PITR) [0] Users and admins make mistakes, and accidentally delete or overwrite data. With PITR you can restore in a new environment, just before the mistake occurred and recover the d…

What I meant is that the archive_command is run only when a WAL segment is completed or when archive_timeout is reached. In the meantime, nothing is backed up. On a low traffic database, this can be a problem. I'm wondering if there is a way to continuously stream the WAL to an object storage like S3, without waiting to have a complete segment.