Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A reverse-delta backup strategy – obvious idea or bad idea?

news.ycombinator.com

Post reply on HN
Print view View original

21–27 of 27 posts

Re: A reverse-delta backup strategy – obvious idea or bad idea?

#21

Post by codingdave »

datastack wrote: post #14
codingdave wrote: post #4

The low likelihood / high impact edge case this does not handle is: "Oops, our data center blew up." An extreme scenario, but one that this method does not handle. It instead turns your most recent backup into a single point of failure because you cannot restore from other backups.

This sounds more like a downside of single site backups

Totally. Which is exactly what your post outlines. You said it yourself: "Only one full copy is needed." You would need to update your logic to have a 2nd copy pushed offsite at some point if you wanted to resolve this edge case.

Re: A reverse-delta backup strategy – obvious idea or bad idea?

#22

Post by sandreas »

datastack wrote: post #16
wmf wrote: post #9

Earlier quoted context omitted.

"For files that changed or were deleted: move them into a new delta folder. For new/changed files: copy them into the latest snapshot folder." is just redneck copy-on-write. It's the same result but less efficient under the hood.

Nice to realize that this boils down to copy on write. Makes it easier to explain.

Is there a reason NOT to use ZFS or BTRFS?

I mean the idea sounds cool but what are you missing? ZFS even works on Windows these days and with tools like zrepl you can configure time based snapshotting, auto-sync and auto-cleanup

Re: A reverse-delta backup strategy – obvious idea or bad idea?

#24

Post by trod1234 »

datastack wrote: post #12
compressedgas wrote: post #2

It works. Already implemented: https://rdiff-backup.net/ https://github.com/rdiff-backup/rdiff-backup There are also other tools which have implemented reverse incremental backup or backup with reverse deduplication which store the most recent backup in contiguous form and fragment the older backups.

Thank you for bringing this to my attention. Knowing that there is a working product using this approach gives me confidence. I'm working on a simple backup app for my personal/family use, so good to know I'm not heading in the wrong direction

These type of projects can easily get sidetracked without a overarching goal. Are you looking to do something specific?

An app (that requires remote infrastructure), seems a bit overkill and if your going through the hassle of doing that you might as well set up the equivalent of what MS used to call the Modern Desktop Experience which is how many enterprise level customers have their systems configured now.

The core parts are cloud-based IDp, storage, and a slipstreamed deployment image which with network connectivity will pull down the config and sets the desired state, replicating the workspace down as needed (with OneDrive).

Backup data layout/strategy/BCDR plan can then be automated from the workspace/IDp/cloud-storage backend with no user interaction/learning curve.

If hardware fails, you use the deployment image to enroll new hardware, login and replicate the user related state down, etc. Automation for recurring tasks can be matched up to the device lifecycle phases (Provision, Enrollment, Recovery, Migration, Retirement). This is basically done in a professional setup with EntraID/Autopilot MDM with MSO365 plans. You can easily set up equivalents but you have to write your own glue.

Most of that structure was taken from Linux grey beards ages ago, MS just made a lot of glue and put it in a nice package.

Re: A reverse-delta backup strategy – obvious idea or bad idea?

#25

Post by brudgers »

In principle, deleting archived data is the opposite of backing up.

It is not clear what problem with existing backup strategies this solves.

I mean you can use a traditional delta backup tool and make one full copy of the current data separately with less chance for errors.

It seems too clever by half and it is not clear to me from the question what problem it solves. Good luck.

Re: A reverse-delta backup strategy – obvious idea or bad idea?

#27

Post by vrighter »

datastack wrote: post #18
vrighter wrote: post #17

I used to work on backup software. Our first version did exactly that. It was a selling point. We later switched approach to a deduplication based one.

Exciting! Yes, the deduplicated approach is superior, if you can accept requiring dedicated software to read the data or can rely on a file system that supports it (like Unix with hard links). I'm looking for a cross-platform solution that is simple and can restore files without any app (in case I didn't maintain my app for the next twenty years). I'm curious if the software you were working on used proprietary forma…

The deduplication in the product I worked on was implemented by me and a colleague of mine, in a custom format. The point of it was to do inline deduplication on a best-effort basis. I.e. handling the case where the system does NOT have enough memory to store hashes for every single block. This might have resulted in some duplicated data if you didn't have enough memory, instead of slowed down to a crawl by hitting the disk (spinning rust, at the time) for each block we wanted to deduplicate.
Post reply on HN