Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Correct Backups Require Filesystem Snapshots

cyounkins.medium.com

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Correct Backups Require Filesystem Snapshots

#31

Post by jjnoakes »

compsciphd wrote: post #27
jjnoakes wrote: post #25

Earlier quoted context omitted.

Not usually. A well-written application that cares about your data should be written in a way to survive sudden power loss, and to such an application, a file-system-level snapshot taken at an arbitrary point in time looks basically the same as sudden power loss. Now, that being said, if you have the ability to set up backups where you can minimize the file system activity, you might be slightly better off doing it,…

that's not really true. there's a reason VSS exists on windows.

Would you mind elaborating? I'm not a Windows expert so I don't know off the top of my head what VSS provides that helps poorly written applications survive sudden power failure more robustly than without VSS in the picture, and how that's better than a file system point-in-time copy-on-write snapshot.

Re: Correct Backups Require Filesystem Snapshots

#32

Post by [deleted] »

ozim wrote: post #19
cyounkins wrote: post #10

Earlier quoted context omitted.

Sure, I've been accused of being overly concerned with correctness. :-) > because imagine doing full snapshot of 1 Terabyte drive it takes I suppose at least an hour anyway. On copy-on-write filesystems, snapshots are nearly instantaneous. > time than setting up and keeping file system snapshots running As shown in the code snippet, pretty much all it takes is a few snapshot commands around the backup command and cha…

Different people care about different things :) I also think about snapshots differently. For me snapshot is actual copy stored on a different hard drive or other medium. So copy operation is never going to be instant. Scenario in the article is power loss so that is also not something I worry about that much. Mostly I worry about hardware failure where I would not be able to turn on my laptop/server again. For power…

[deleted]

Re: Correct Backups Require Filesystem Snapshots

#34

Post by AshamedCaptain »

compsciphd wrote: post #26

even with file system snapshots, your backup can be corrupt. the example of the browser profile is a classic case. imagine multiple writes occur and all of them have to occur for the profile to be correct (either multiple files are being written or firefox will write multiple blocks to the file). if the snapshot operation occurs in the middle, then the snapshot will be "corrupt". I believe this is actually the entire…

The worst part is that the article itself says this. This is exactly like "pulling the power on the running system" and then imaging the hard disk. Even if it happens to work, that is not really a good idea.

Re: Correct Backups Require Filesystem Snapshots

#35

Post by [deleted] »

ozim wrote: post #6

Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at t…

[deleted]

Re: Correct Backups Require Filesystem Snapshots

#36

Post by liuliu »

kazinator wrote: post #20
liuliu wrote: post #5

Actually, I have a hard time to understand this. While snapshot reduces file corruptions, it is not a guarantee the best I understand. A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption). Thus, any discussion on file corruptions without a…

A snapshot captures some state of the filesystem that the filesystem actually had at some point. So when you recover that state, and then run the application, it's similar to the OS having crashed or the power having been lost. Whereas non-snapshot backups don't have that property; the material in the backup is not necessarily identical to any past state of the filesystem that existed. It's something like a past stat…

I think we need to be a bit more concrete than analogies for my brain to process. So here we go:

If your application relies on multi-file collaborated persisted states (for example, an append-only log and a database snapshot), application can make reasonable assumptions on when a file state is committed, such as a database snapshot is `fsync`ed before the append-only log started. Ideally, even on a filesystem without transaction support, this order is preserved in time.

However, it may not be preserved for a "file-to-file" backup system because it can loop over the append-only log file first before the database snapshot, causing ordering issues. That could result a "ABA" problem, where the append-only log is corresponding to an older database snapshot, and potentially causes issues.

That has been said, is this a common scenario (multi-file persisted state) for applications? (I believe SQLite handles this particular ordering issue fine). I am not sure, and just want to call out filesystem snapshot solves a very particular problem.

If your application relies only one file for the state, or these files are orthogonal to each other (.xlsx or .docx or any .markdown, .mp4, .jpg files), it is a non-issue. And if you need to backup a database, you better do that with the database provided tools.

Re: Correct Backups Require Filesystem Snapshots

#37

Post by [deleted] »

ozim wrote: post #6

Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at t…

[deleted]

Re: Correct Backups Require Filesystem Snapshots

#38

Post by kazinator »

liuliu wrote: post #36
kazinator wrote: post #20

Earlier quoted context omitted.

A snapshot captures some state of the filesystem that the filesystem actually had at some point. So when you recover that state, and then run the application, it's similar to the OS having crashed or the power having been lost. Whereas non-snapshot backups don't have that property; the material in the backup is not necessarily identical to any past state of the filesystem that existed. It's something like a past stat…

I think we need to be a bit more concrete than analogies for my brain to process. So here we go: If your application relies on multi-file collaborated persisted states (for example, an append-only log and a database snapshot), application can make reasonable assumptions on when a file state is committed, such as a database snapshot is `fsync`ed before the append-only log started. Ideally, even on a filesystem without…

I think it's not just between files, but within a file.

The backup process opens some N byte file and starts copying it. Some portion of the file is backed up until byte K. At that point the application writes a transaction whereby some of the data is written above K, and some below K. The backup continues and backs up half the transaction above K, combining that with the old data below K before the transaction happened.

If the application structures its scattered write transactions such that they write in increasing offset order, then maybe it's okay.

Re: Correct Backups Require Filesystem Snapshots

#39

Post by viraptor »

notacoward wrote: post #23

Should clarify that OP is talking about a consumer use case. In the enterprise space, quiescing applications and databases during a snapshot was standard at least as far back as 1992 when I started working on HA software which would (among its other functions) automate the process. For quite a few years after that, I worked on systems where block-level snapshots were more common (and performed better) than filesystem…

For quite a few years after 1992 we didn't really have simple solutions that could do atomic snapshots. I'm not sure about specialised enterprise cases, but for Linux consumers, the first one would be lvm right? (And that was not write-barrier-safe until 2010)

Re: Correct Backups Require Filesystem Snapshots

#40

Post by grdomzal »

compsciphd wrote: post #26

even with file system snapshots, your backup can be corrupt. the example of the browser profile is a classic case. imagine multiple writes occur and all of them have to occur for the profile to be correct (either multiple files are being written or firefox will write multiple blocks to the file). if the snapshot operation occurs in the middle, then the snapshot will be "corrupt". I believe this is actually the entire…

> I believe this is actually the entire point of Windows' volume shadow service (which is sort of poo-pooed in the article), to enable applications to tell the snapshot mechanism "wait, I'm in the middle of a file system transaction" and then to pause writes until the snapshot operation occurs after they finish the in process transaction.

Former maintainer of VSS here. Yes, that's exactly right. In fact, filesystem snapshots are usually not enough for true application level consistency. As others have noted elsewhere, a filesystem snapshot is the equivalent to yanking the power cord out of the back of your computer. It's good, assuming your filesystem does atomic writes / copy-on-write / write-to-new. But we can do even better.

Imagine I have two databases - one traditional relational DB storing my app content, and a second log database. You want to keep these in sync. Well, good luck doing this with the filesystem alone. Usually your DBMS will need to be involved as well, and this is where the VSS "writer" concept comes in. When a snapshot is being taken, applications such as SQL will be invited to participate. Typically, this means they'll start to hold up writes so that things will be quieter for the snapshot. But they will also have a chance, after the fact, and to actually clean up the snapshot itself. In this case, the DBMS could roll back the log database to then match the content database.

It's correct that NTFS doesn't support snapshots natively, but Windows has the volsnap.sys driver that takes care of it. For all intents and purposes, NTFS does support copy-on-write snapshots.

Complicated? Sure. But it was quite capable, and actually a pretty cool (but sadly under appreciated) piece of technology.

Post reply on HN