Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing WAL-G: Faster Disaster Recovery for Postgres

citusdata.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Introducing WAL-G: Faster Disaster Recovery for Postgres

#51

Post by trojanowski »

upbeatlinux wrote: post #42
fdr wrote: post #36

Earlier quoted context omitted.

Do you want GPG based or some other client side encryption, or S3's encryption support? The latter could probably just be turned on. The former is a feature requiring code.

Ideally the presence of the `WALE_GPG_KEY_ID` env var should enable encrypted backups https://github.com/wal-e/wal-e#encryption . Put differently to be a "successor" it needs to be a drop in replacement ;)

Alternatively you could use a combination of AES and RSA similar how pghoard implements it: https://github.com/ohmu/pghoard

The RSA keys (or path to them) would be passed as environment variables. It would be a little easier to setup than gpg (especially for automatic backup restoration).

Re: Introducing WAL-G: Faster Disaster Recovery for Postgres

#52

Post by ngrilly »

daurnimator wrote: post #49
ngrilly wrote: post #44

Earlier quoted context omitted.

What I meant is that the archive_command is run only when a WAL segment is completed or when archive_timeout is reached. In the meantime, nothing is backed up. On a low traffic database, this can be a problem. I'm wondering if there is a way to continuously stream the WAL to an object storage like S3, without waiting to have a complete segment.

S3 is a block store; not something you can really stream to. However it might be interesting to stream WAL logs to e.g. AWS Kinesis....

You're right. S3 is an object store and doesn't support the append operation, which is required for what I want to do. Thanks!

Re: Introducing WAL-G: Faster Disaster Recovery for Postgres

#53

Post by simtel20 »

daurnimator wrote: post #49
ngrilly wrote: post #44

Earlier quoted context omitted.

What I meant is that the archive_command is run only when a WAL segment is completed or when archive_timeout is reached. In the meantime, nothing is backed up. On a low traffic database, this can be a problem. I'm wondering if there is a way to continuously stream the WAL to an object storage like S3, without waiting to have a complete segment.

S3 is a block store; not something you can really stream to. However it might be interesting to stream WAL logs to e.g. AWS Kinesis....

You can open multi-part transfers and close out the transfer when you're ready, which can be used so that it is very close to streaming; for this case perhaps it's close enough to try with wall-g if it otherwise supports it.

Re: Introducing WAL-G: Faster Disaster Recovery for Postgres

#55

Post by justinclift »

brianwawok wrote: post #24
kafkes wrote: post #18

Earlier quoted context omitted.

WAL-G is not yet production ready, but it has been used in a staging environment for the past few weeks without any issues. Once fdr adds parallel WAL support, he plans to take it into production.

Cool cool. is google cloud storage on the roadmap?

WAL-G looks like it should be able to talk to Minio (minio.io, S3 compatible thing, also written in go) as a backend instead of S3 itself.

Minio has an interesting feature where it can be a "gateway" to other cloud storage. Google Cloud Storage is one of their specific examples:

https://docs.minio.io/docs/minio-gateway-for-gcs

So WAL-G would talk to Minio, and Minio would transparently proxy that to GCS.

Re: Introducing WAL-G: Faster Disaster Recovery for Postgres

#56

Post by boulos »

brianwawok wrote: post #24
kafkes wrote: post #18

Earlier quoted context omitted.

WAL-G is not yet production ready, but it has been used in a staging environment for the past few weeks without any issues. Once fdr adds parallel WAL support, he plans to take it into production.

Cool cool. is google cloud storage on the roadmap?

There was some mention about resumable uploads in the blogpost which sadly each provider handles differently (that is the GCS layer that supports the S3 API does not accept resumable uploads).

Disclosure: I work on Google Cloud (so I'd love to see this tool point at GCS).

Re: Introducing WAL-G: Faster Disaster Recovery for Postgres

#57

Post by boulos »

fdr wrote: post #31
jfkw wrote: post #23

Will WAL-G eventually support the same archive targets as WAL-E (S3 and work-alikes, Azure Blob Store, Google Storage, Swift, File System)?

As I'm probably the steward on this going forward: unknown. I don't intend to implement them unless I need them. Would I take a patch with good coverage that implemented those.

Would you be willing to own the abstraction for multiple backends? The code is currently only a bit hardcoded to S3/AWS, but I assume most of the "work" will be discussing how to abstract different transports, exponential backoff, resumable uploads, and so on.

Fwiw, the GCS client for go (import "cloud.google.com/go/storage") is very straightforward. Though as others have pointed out, it might be worthwhile to just try to use minio-go if you want to gain Ceph as well.

Disclosure: I work on Google Cloud (and if the way is paved, we will contribute here; seems like a great project)

Re: Introducing WAL-G: Faster Disaster Recovery for Postgres

#58

Post by bpicolo »

paulsutter wrote: post #41
ozgune wrote: post #33

Earlier quoted context omitted.

Hey Dan, nice to hear from you! > our prod base backups take 36 hours to restore, so if the recovery speed improvements are as advertised, that's a big win. Yes, if you attach 16TB of storage to each instance, your back-up restores may take a while. :))

Is it possible to run a continuous restore in parallel with normal operation so that there's a warm standby (almost) ready to go? Especially in another data center?

Yeah, just have a read replica. Can promote it to master later

Re: Introducing WAL-G: Faster Disaster Recovery for Postgres

#59

Post by brianwawok »

justinclift wrote: post #55
brianwawok wrote: post #24

Earlier quoted context omitted.

Cool cool. is google cloud storage on the roadmap?

WAL-G looks like it should be able to talk to Minio (minio.io, S3 compatible thing, also written in go) as a backend instead of S3 itself. Minio has an interesting feature where it can be a "gateway" to other cloud storage. Google Cloud Storage is one of their specific examples: https://docs.minio.io/docs/minio-gateway-for-gcs So WAL-G would talk to Minio, and Minio would transparently proxy that to GCS.

Neat. My concern out of the gate is what would be the perf hit.

I assume I am switching from WAL-E to WAL-G for more perf. But WAL-E speaks GCS. If WAL-G needs an extra hop to do so, may lose some of the point of it..

Re: Introducing WAL-G: Faster Disaster Recovery for Postgres

#60

Post by justinclift »

brianwawok wrote: post #59
justinclift wrote: post #55

Earlier quoted context omitted.

WAL-G looks like it should be able to talk to Minio (minio.io, S3 compatible thing, also written in go) as a backend instead of S3 itself. Minio has an interesting feature where it can be a "gateway" to other cloud storage. Google Cloud Storage is one of their specific examples: https://docs.minio.io/docs/minio-gateway-for-gcs So WAL-G would talk to Minio, and Minio would transparently proxy that to GCS.

Neat. My concern out of the gate is what would be the perf hit. I assume I am switching from WAL-E to WAL-G for more perf. But WAL-E speaks GCS. If WAL-G needs an extra hop to do so, may lose some of the point of it..

Yeah, no idea personally. Haven't used the gateway functionality in Minio at all.

That being said, the Minio team seem pretty good with writing performance optimised code. Frank Wessels (on Minio team), has been writing articles about Go assembler and other Go optimisation things recently. eg:

• https://blog.minio.io/accelerating-blake2b-by-4x-using-simd-...

• https://blog.minio.io/golang-internals-part-2-nice-benefits-...

So the performance impact might not be such a problem. :)

Post reply on HN