Zfsbackrest: Pgbackrest style encrypted backups for ZFS filesystems
1–10 of 13 posts
Re: Zfsbackrest: Pgbackrest style encrypted backups for ZFS filesystems
#2Finally! Been looking for this a long time. File-based backups for large Pg databases are not very scalable (even incremental), having this in my toolkit would be amazing.
Re: Zfsbackrest: Pgbackrest style encrypted backups for ZFS filesystems
#3Finally! Been looking for this a long time. File-based backups for large Pg databases are not very scalable (even incremental), having this in my toolkit would be amazing.
I'm not sure I follow. Wouldn't this be file-based (zfs-dataset-based) incremental backups? I don't think this has anything to do with postgresql other than copying the style of pgBackRest.
Re: Zfsbackrest: Pgbackrest style encrypted backups for ZFS filesystems
#4Finally! Been looking for this a long time. File-based backups for large Pg databases are not very scalable (even incremental), having this in my toolkit would be amazing.
I'm not sure I follow. Wouldn't this be file-based (zfs-dataset-based) incremental backups? I don't think this has anything to do with postgresql other than copying the style of pgBackRest.
this uses `zfs send @snapshot` which is block-level, not file-level.
Re: Zfsbackrest: Pgbackrest style encrypted backups for ZFS filesystems
#5Earlier quoted context omitted.
I'm not sure I follow. Wouldn't this be file-based (zfs-dataset-based) incremental backups? I don't think this has anything to do with postgresql other than copying the style of pgBackRest.
this uses `zfs send @snapshot` which is block-level, not file-level.
Oh! So the issue with large postgres backups is the number of files?
Re: Zfsbackrest: Pgbackrest style encrypted backups for ZFS filesystems
#6Earlier quoted context omitted.
this uses `zfs send @snapshot` which is block-level, not file-level.
Oh! So the issue with large postgres backups is the number of files?
No. Postgres data files are 1Gb each. When you change just one byte in a table, a whole 1Gb file gets updated (just 1 byte change, effectively). Your file-based backup tool now has to upload 1Gb of data to save 1 byte of actual changes.
Re: Zfsbackrest: Pgbackrest style encrypted backups for ZFS filesystems
#7Earlier quoted context omitted.
Oh! So the issue with large postgres backups is the number of files?
No. Postgres data files are 1Gb each. When you change just one byte in a table, a whole 1Gb file gets updated (just 1 byte change, effectively). Your file-based backup tool now has to upload 1Gb of data to save 1 byte of actual changes.
They fixed that in pgbackrest a while ago: https://pgbackrest.org/user-guide.html#backup/block
It was a major pain point for my backups for years.
Re: Zfsbackrest: Pgbackrest style encrypted backups for ZFS filesystems
#8Earlier quoted context omitted.
No. Postgres data files are 1Gb each. When you change just one byte in a table, a whole 1Gb file gets updated (just 1 byte change, effectively). Your file-based backup tool now has to upload 1Gb of data to save 1 byte of actual changes.
They fixed that in pgbackrest a while ago: https://pgbackrest.org/user-guide.html#backup/block It was a major pain point for my backups for years.
Does that work with S3, etc.? I don't remember them allowing partial file uploads.
Re: Zfsbackrest: Pgbackrest style encrypted backups for ZFS filesystems
#9Earlier quoted context omitted.
They fixed that in pgbackrest a while ago: https://pgbackrest.org/user-guide.html#backup/block It was a major pain point for my backups for years.
Does that work with S3, etc.? I don't remember them allowing partial file uploads.
I believe so, because it is done in conjunction with their file bundling feature and doesn't rely on support from the blob storage backend.
They create a new file with the diffs of a bundle of Postgres files, and upload that to blob storage.
Re: Zfsbackrest: Pgbackrest style encrypted backups for ZFS filesystems
#10This seems to store the zfs send stream. That's a bad idea.
> Incremental ZFS send streams do not have any of these properties and full ZFS send streams only have a few of them. Neither full nor incremental streams have any resilience against damage to the stream; a stream is either entirely intact or it's useless. Neither has selective restores or readily available indexes. Incremental streams are completely useless without everything they're based on. All of these issues will sooner or later cause you pain if you use ZFS streams as a backup format.
https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSSendNotA...