Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PGHoard: Tools for making PostgreSQL backups to cloud object storages

blog.aiven.io

Post reply on HN
Print view View original

1–10 of 14 posts

Re: PGHoard: Tools for making PostgreSQL backups to cloud object storages

#3

Post by brudgers »

melor wrote: post #2

Takes care of realtime WAL streaming, compression, encryption, restoration and backup expiration among other things. Open Source and written in Python.

Curious if it backs up to other cloud storage providers in vendor neutral ways.

Re: PGHoard: Tools for making PostgreSQL backups to cloud object storages

#4

Post by melor »

brudgers wrote: post #3
melor wrote: post #2

Takes care of realtime WAL streaming, compression, encryption, restoration and backup expiration among other things. Open Source and written in Python.

Curious if it backs up to other cloud storage providers in vendor neutral ways.

Currently S3 (AWS + compatible), Google Cloud, OpenStack Swift, Azure (experimental), local disk and Ceph (via S3 or Swift) are supported. More can be added quite easily as the object storage logic is behind an extendable interface.

Which vendor neutral protocol are you interested in using?

Re: PGHoard: Tools for making PostgreSQL backups to cloud object storages

#5

Post by oskari »

brudgers wrote: post #3
melor wrote: post #2

Takes care of realtime WAL streaming, compression, encryption, restoration and backup expiration among other things. Open Source and written in Python.

Curious if it backs up to other cloud storage providers in vendor neutral ways.

PGHoard has quite high unit test coverage (85%) and it's pretty easy to add a new object storage configuration to tests to verify that all the APIs used by PGHoard work properly.

Re: PGHoard: Tools for making PostgreSQL backups to cloud object storages

#7

Post by melor »

waffle_ss wrote: post #6

How does this compare to WAL-E? https://github.com/wal-e/wal-e

Both do mostly the same thing with some differences. The biggest difference currently could be that WAL-E uses the PostgreSQL "archive_command" to send incremental backups (WAL files) in complete 16 megabyte chunks, whereas PGHoard uses real-time streaming with "pg_receivexlog", making the data loss window much smaller in case of a disaster.

Re: PGHoard: Tools for making PostgreSQL backups to cloud object storages

#9

Post by melor »

anarazel wrote: post #8

Do you prevent segments from being removed while they're not yet received by pg_receivexlog? wal_keep_segments or replication slots?

A replication slot can be used by defining it in the pghoard.json configuration. However, the slot needs to be created (and removed after no longer needed, important!) manually. We've been planning to add more automatic replication slot management to PGHoard.

Re: PGHoard: Tools for making PostgreSQL backups to cloud object storages

#10

Post by merb »

melor wrote: post #4
brudgers wrote: post #3

Earlier quoted context omitted.

Curious if it backs up to other cloud storage providers in vendor neutral ways.

Currently S3 (AWS + compatible), Google Cloud, OpenStack Swift, Azure (experimental), local disk and Ceph (via S3 or Swift) are supported. More can be added quite easily as the object storage logic is behind an extendable interface. Which vendor neutral protocol are you interested in using?

What will happen when the Storage (swift or ceph) is offline for some time?
Post reply on HN