Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Correct Backups Require Filesystem Snapshots

cyounkins.medium.com

Post reply on HN
Print view View original

11–20 of 78 posts

Re: Correct Backups Require Filesystem Snapshots

#11

Post by aidenn0 »

michaelt wrote: post #4
drewzero1 wrote: post #3

Earlier quoted context omitted.

For a long time I didn't know where to start with backups, because I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.) Eventually I realized that some backups are better than no backups at all, and prioritized getting _something_ in place to at the very least have copies of important files copied regularly to another drive somewhere.

> I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.) It depends a lot on your use case. If you're backing up a production database system that's in constant use, then the files are almost certain to change while the backup is in progress. And a backup of everything on the database server except the data probably isn't what you want! On the other h…

It's been a while, but with e.g. Oracle 10, it was specifically stated in the docs that backing up the data-file(s), followed by the archive-logs, you would end up with a recoverable backup. I'm not sure if other databases are designed that way.

Re: Correct Backups Require Filesystem Snapshots

#12

Post by rzzzt »

liuliu wrote: post #5

Actually, I have a hard time to understand this. While snapshot reduces file corruptions, it is not a guarantee the best I understand. A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption). Thus, any discussion on file corruptions without a…

The Volume Shadow Copy Service mentioned at the end has support for application-level notifications via writers, the most prominent one being SQL Writer: https://docs.microsoft.com/en-us/sql/relational-databases/ba...

Re: Correct Backups Require Filesystem Snapshots

#14

Post by adrianmonk »

liuliu wrote: post #5

Actually, I have a hard time to understand this. While snapshot reduces file corruptions, it is not a guarantee the best I understand. A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption). Thus, any discussion on file corruptions without a…

One way of looking at it is that applications already need to be able to survive power loss, a crash of the computer, a crash of the application, or a forcible kill of the application. If they can survive those, they can probably survive a filesystem snapshot being taken at just the wrong moment.

Re: Correct Backups Require Filesystem Snapshots

#16

Post by forgotmypw17 »

ozim wrote: post #6

Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at t…

> I don't need to backup my Discord

If you don't back up your Discord, what will you do when Discord changes how it works or stops working?

Re: Correct Backups Require Filesystem Snapshots

#17

Post by ComputerGuru »

aborsy wrote: post #15

Ubuntu supports ZFS with one click at installation time. You can snapshot your system every 15 minutes or as you desire.

Note that it’s not really a supported option and was supposed to be removed from the installer for the latest release but that was cancelled because there wasn’t enough time to do so before the freeze. It’s certain to be dropped in the next release.

So by all means, play with it but don’t consider it to be stable.

https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1966...

https://code.launchpad.net/~ubuntu-installer/ubiquity/+git/u...

Edit: fixed link.

Re: Correct Backups Require Filesystem Snapshots

#19

Post by ozim »

cyounkins wrote: post #10
ozim wrote: post #6

Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at t…

Sure, I've been accused of being overly concerned with correctness. :-) > because imagine doing full snapshot of 1 Terabyte drive it takes I suppose at least an hour anyway. On copy-on-write filesystems, snapshots are nearly instantaneous. > time than setting up and keeping file system snapshots running As shown in the code snippet, pretty much all it takes is a few snapshot commands around the backup command and cha…

Different people care about different things :)

I also think about snapshots differently.

For me snapshot is actual copy stored on a different hard drive or other medium. So copy operation is never going to be instant.

Scenario in the article is power loss so that is also not something I worry about that much. Mostly I worry about hardware failure where I would not be able to turn on my laptop/server again. For power loss UPS or battery in laptop is my go to solution.

Re: Correct Backups Require Filesystem Snapshots

#20

Post by kazinator »

liuliu wrote: post #5

Actually, I have a hard time to understand this. While snapshot reduces file corruptions, it is not a guarantee the best I understand. A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption). Thus, any discussion on file corruptions without a…

A snapshot captures some state of the filesystem that the filesystem actually had at some point. So when you recover that state, and then run the application, it's similar to the OS having crashed or the power having been lost.

Whereas non-snapshot backups don't have that property; the material in the backup is not necessarily identical to any past state of the filesystem that existed. It's something like a past state plus random roll-backs of files, or portions of files, from multiple previous states.

Guess which of these situations applications are much more likely to be able to recover from (if any at all)?

When people write crash recovery code, they typically assume that the world simply stopped, not that it stopped, and then some data was randomly rewound to unspecified older states.

I don't think that you can reasonably defend against data that is restored from a backup, where the oldest part of the backup is an hour older than the newest.

A backup is like a raster scan image of a fast moving object. What should be a rectangular train car looks like a parallelogram: it's not a picture of any scene that existed. Imagine that the raster lines are randomly sampled (not a progressive scan, or even interlaced) and now recover a sane image.

Post reply on HN