Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Correct Backups Require Filesystem Snapshots

cyounkins.medium.com

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Correct Backups Require Filesystem Snapshots

#21

Post by kazinator »

adrianmonk wrote: post #14
liuliu wrote: post #5

Actually, I have a hard time to understand this. While snapshot reduces file corruptions, it is not a guarantee the best I understand. A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption). Thus, any discussion on file corruptions without a…

One way of looking at it is that applications already need to be able to survive power loss, a crash of the computer, a crash of the application, or a forcible kill of the application. If they can survive those, they can probably survive a filesystem snapshot being taken at just the wrong moment.

Right! But not necessarily a file by file backup.

Re: Correct Backups Require Filesystem Snapshots

#22

Post by ozim »

forgotmypw17 wrote: post #16
ozim wrote: post #6

Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at t…

> I don't need to backup my Discord If you don't back up your Discord, what will you do when Discord changes how it works or stops working?

Important part of Discord are people - having backup means of communication on different platforms is best I think.

Re: Correct Backups Require Filesystem Snapshots

#23

Post by notacoward »

Should clarify that OP is talking about a consumer use case. In the enterprise space, quiescing applications and databases during a snapshot was standard at least as far back as 1992 when I started working on HA software which would (among its other functions) automate the process. For quite a few years after that, I worked on systems where block-level snapshots were more common (and performed better) than filesystem-level. Filesystem snapshots are great, and it's also great to warn people about the dangers of doing backups without some sort of protection against in-flight changes, but the title as written isn't quite correct.

Re: Correct Backups Require Filesystem Snapshots

#24

Post by dsr_ »

ozim wrote: post #19
cyounkins wrote: post #10

Earlier quoted context omitted.

Sure, I've been accused of being overly concerned with correctness. :-) > because imagine doing full snapshot of 1 Terabyte drive it takes I suppose at least an hour anyway. On copy-on-write filesystems, snapshots are nearly instantaneous. > time than setting up and keeping file system snapshots running As shown in the code snippet, pretty much all it takes is a few snapshot commands around the backup command and cha…

Different people care about different things :) I also think about snapshots differently. For me snapshot is actual copy stored on a different hard drive or other medium. So copy operation is never going to be instant. Scenario in the article is power loss so that is also not something I worry about that much. Mostly I worry about hardware failure where I would not be able to turn on my laptop/server again. For power…

The snapshot takes a second to generate; then you can take as long as your hardware needs to get it somewhere else.

Re: Correct Backups Require Filesystem Snapshots

#25

Post by jjnoakes »

throw7 wrote: post #13

Wouldn't you need to quiesce the db or application before the snapshot?

Not usually. A well-written application that cares about your data should be written in a way to survive sudden power loss, and to such an application, a file-system-level snapshot taken at an arbitrary point in time looks basically the same as sudden power loss.

Now, that being said, if you have the ability to set up backups where you can minimize the file system activity, you might be slightly better off doing it, but the ROI is probably fairly low unless it's extremely trivial to set that up for everything that's running.

Re: Correct Backups Require Filesystem Snapshots

#26

Post by compsciphd »

even with file system snapshots, your backup can be corrupt.

the example of the browser profile is a classic case.

imagine multiple writes occur and all of them have to occur for the profile to be correct (either multiple files are being written or firefox will write multiple blocks to the file). if the snapshot operation occurs in the middle, then the snapshot will be "corrupt".

I believe this is actually the entire point of Windows' volume shadow service (which is sort of poo-pooed in the article), to enable applications to tell the snapshot mechanism "wait, I'm in the middle of a file system transaction" and then to pause writes until the snapshot operation occurs after they finish the in process transaction.

without such a mechanism, you are always going to be at risk with snapshots.

In https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf we avoided this problem by combining 2 mechanisms without having modifying applications with such a service

1) we used a log structured file system (that was inherently a snapshot, every log entry was individually mountable) and 2) we used a checkpoint/restart mechanism that saved process state and enabled us to restart the processes combined with the file system state as it was at checkpoint time. (Checkpoint would also sync all dirty pages to disk and that fs state after the sync is what we tied to the checkpoint state).

So when a process would be resumed, the file system would look exactly as the process expected it, even if the process was in the middle of what can be referred to a transaction. But that only worked because the processes were restored along with file system, if we only restored the file system, it could be inconsistent.

Re: Correct Backups Require Filesystem Snapshots

#27

Post by compsciphd »

jjnoakes wrote: post #25
throw7 wrote: post #13

Wouldn't you need to quiesce the db or application before the snapshot?

Not usually. A well-written application that cares about your data should be written in a way to survive sudden power loss, and to such an application, a file-system-level snapshot taken at an arbitrary point in time looks basically the same as sudden power loss. Now, that being said, if you have the ability to set up backups where you can minimize the file system activity, you might be slightly better off doing it,…

that's not really true. there's a reason VSS exists on windows.

Re: Correct Backups Require Filesystem Snapshots

#28

Post by SoftTalker »

blibble wrote: post #18
throw7 wrote: post #13

Wouldn't you need to quiesce the db or application before the snapshot?

yep, otherwise you risk some state being in memory and not on the disk

Exactly. A snapshot is a point in time on the storage media. It allows you to make a backup as if the backup occurred at that point in time, i.e. files are not changing while the backup is running. A snapshot is not a guarantee that any particular file is consistent with the state of the application that is using it. It's still a backup of an "open" file with all the potential issues that implies.

Re: Correct Backups Require Filesystem Snapshots

#29

Post by kroltan »

forgotmypw17 wrote: post #16
ozim wrote: post #6

Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at t…

> I don't need to backup my Discord If you don't back up your Discord, what will you do when Discord changes how it works or stops working?

The application is self-modifying and partially server-defined, as well as being an online communication tool. Discord-the-company can change it at basically any time, and disallow any versions deemed "inappropriate".

Re: Correct Backups Require Filesystem Snapshots

#30

Post by LanternLight83 »

ozim wrote: post #6

Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at t…

These arguments apply to external backups, but if you use a filesystem like ZFS or BTRFS, than snapshots are atomic, essentially instant, and can even be sent over the network as diffs/deltas against the last snapshot that was sent, so backing up a TB drive over the network every hour is total reasonable. These filesystems also give you access to snapshots via the FS, so you don't need to restore a snapshot to access your files, it's as easy mounting the readonly-snapshots directory, which you could even just keep mounted 24/7, making deleted files just a `cd` away, or quicker with a shell script that jumps first into the last snapshot of your current directory and then recursively back to the last one before that.

Not that I've gotten around to writing that script for myself :c

But the automatic snapshots are much easier c:

Post reply on HN