Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Correct Backups Require Filesystem Snapshots

cyounkins.medium.com

Post reply on HN
Print view View original

71–78 of 78 posts

Re: Correct Backups Require Filesystem Snapshots

#71

Post by blibble »

topspin wrote: post #48
blibble wrote: post #18

Earlier quoted context omitted.

yep, otherwise you risk some state being in memory and not on the disk

Your answer is too definitive. Applications and databases can indeed suffer power loss while losing nothing of value. An ACID database used by a correctly designed application will lose nothing on power loss; state "in memory" is uncommitted in such a system effectively doesn't exist. It never happened and, furthermore, no one cares. The term of art is "crash consistent," and any ACID database must preserve all commi…

this is fantastic and all for carefully written databases (none of which have ever had any bugs)

unfortuntely most applications aren't like that and assume the filesystem is both atomic and durable

Re: Correct Backups Require Filesystem Snapshots

#72

Post by topspin »

blibble wrote: post #71
topspin wrote: post #48

Earlier quoted context omitted.

Your answer is too definitive. Applications and databases can indeed suffer power loss while losing nothing of value. An ACID database used by a correctly designed application will lose nothing on power loss; state "in memory" is uncommitted in such a system effectively doesn't exist. It never happened and, furthermore, no one cares. The term of art is "crash consistent," and any ACID database must preserve all commi…

this is fantastic and all for carefully written databases (none of which have ever had any bugs) unfortuntely most applications aren't like that and assume the filesystem is both atomic and durable

> 99%

Your figure is too high. Commercial relational databases, many open source databases and widely used file systems are all sufficient. Oracle, SQL Server, DB/2, PostgreSQL and MySql/MariaDB (with InnoDB) are all ACID systems and recover from crash consistent backups fine. Ext4, XFS, NTFS, etc. are all sufficient file systems, unusually in default form. Reliable storage devices are widely available.

Many applications designed around these platforms use COMMIT properly. When they're don't quiescing the database probably won't help with backups anyhow; all you're doing is deferring writes and if the application is not ACID then you can still lose. The database itself, however, will recover and since we're tossing probabilities around, that's 99% of the battle.

Re: Correct Backups Require Filesystem Snapshots

#73

Post by blibble »

topspin wrote: post #72
blibble wrote: post #71

Earlier quoted context omitted.

this is fantastic and all for carefully written databases (none of which have ever had any bugs) unfortuntely most applications aren't like that and assume the filesystem is both atomic and durable

> 99% Your figure is too high. Commercial relational databases, many open source databases and widely used file systems are all sufficient. Oracle, SQL Server, DB/2, PostgreSQL and MySql/MariaDB (with InnoDB) are all ACID systems and recover from crash consistent backups fine. Ext4, XFS, NTFS, etc. are all sufficient file systems, unusually in default form. Reliable storage devices are widely available. Many applicat…

I'd bet quite a large amount of money that if you could see every process running on every server on earth: less than 1% of the total would be database processes

meanwhile most of the rest seek randomly in files, buffer and write data without calling ever fsync, or append without a pointer and appropriate write barriers (I'm looking at you journald)

Re: Correct Backups Require Filesystem Snapshots

#74

Post by jjnoakes »

tatersolid wrote: post #70
jjnoakes wrote: post #42

Earlier quoted context omitted.

Doesn't sound like that protects against power failure. Most databases I use keep a write-ahead log, which guards against power failure without requiring an external service. Multiple writes are handled without an issue. If power is lost, the database recovers automatically using whatever is on disk, no matter when it was interrupted. So a file system snapshot behaves the same and the database can start up from the s…

In many if not most production databases, that write-ahead log will be on a different volume or file system from the database files. This is often recommended in documentation for performance reasons. So now it is quite possible you back up an older WAL coupled with a newer DB file that has partial writes inside. Boom. This “multi-volume, multi-writer coordination” is the sort of problem VSS solves on windows. I’m no…

In that case you just back up the database first and the WAL second.

Re: Correct Backups Require Filesystem Snapshots

#75

Post by topspin »

blibble wrote: post #73
topspin wrote: post #72

Earlier quoted context omitted.

> 99% Your figure is too high. Commercial relational databases, many open source databases and widely used file systems are all sufficient. Oracle, SQL Server, DB/2, PostgreSQL and MySql/MariaDB (with InnoDB) are all ACID systems and recover from crash consistent backups fine. Ext4, XFS, NTFS, etc. are all sufficient file systems, unusually in default form. Reliable storage devices are widely available. Many applicat…

I'd bet quite a large amount of money that if you could see every process running on every server on earth: less than 1% of the total would be database processes meanwhile most of the rest seek randomly in files, buffer and write data without calling ever fsync, or append without a pointer and appropriate write barriers (I'm looking at you journald)

I don't know what your basis of measurement is, so I'll continue to ignore your ratios. Tolerance of crash consistent state isn't exclusive to databases. Filesystems themselves are often designed to tolerate, for example, sudden power loss without corruption. Applications that manage large collections of files can also achieve this; maildir is an example of such a scheme. All of these systems are using related principles of consistency, and all of them can be correctly backed up by copying volume snapshots.

Re: Correct Backups Require Filesystem Snapshots

#76

Post by blibble »

topspin wrote: post #75
blibble wrote: post #73

Earlier quoted context omitted.

I'd bet quite a large amount of money that if you could see every process running on every server on earth: less than 1% of the total would be database processes meanwhile most of the rest seek randomly in files, buffer and write data without calling ever fsync, or append without a pointer and appropriate write barriers (I'm looking at you journald)

I don't know what your basis of measurement is, so I'll continue to ignore your ratios. Tolerance of crash consistent state isn't exclusive to databases. Filesystems themselves are often designed to tolerate, for example, sudden power loss without corruption. Applications that manage large collections of files can also achieve this; maildir is an example of such a scheme. All of these systems are using related princi…

are you ignoring the main point too, or simply missing it entirely (repeatedly)?

the fact there exist some applications (databases, applications using maildir) that correctly use write barriers doesn't mean all applications are always consistent on disk

Re: Correct Backups Require Filesystem Snapshots

#77

Post by bretpiatt »

jiggawatts wrote: post #50
bretpiatt wrote: post #47

A system snapshot for a personal computer is potentially a really difficult thing to restore (ex. you break the device and need a new one, unless it matches exactly restoring snapshot likely puts you in OS recovery mode). A volume snapshot if you're technical enough to split up OS and application data volumes on your local drive can potentially help but you still have registry or other issues. The best way to handle…

This is 100% false and nobody should ever follow this advice. I use snapshots all the time because Windows uses them automatically to get complete and consistent backups. Snapshots are always better then no snapshots. Full backups are always safer than partial backups. It takes just ONE forgotten path or file to make a backup completely impossible to restore. Don’t be a smart ass with backups. Just don’t.

A full system snapshot is not 100% always the correct solution, data protection is complicated.

An example hypothetical, you have deeply embedded malware and you're unsure when it infected the system, it could have been sitting latent for months. In this situation you have to restore back and lose months of application data or restore a recent snapshot and then try to remove the malware in a clean room. It is much safer and easier to start from a clean OS and then restoring only the data on the system which is much simpler to scan for infection and sanitize.

Re: Correct Backups Require Filesystem Snapshots

#78

Post by bretpiatt »

iggldiggl wrote: post #67
bretpiatt wrote: post #47

A system snapshot for a personal computer is potentially a really difficult thing to restore (ex. you break the device and need a new one, unless it matches exactly restoring snapshot likely puts you in OS recovery mode). A volume snapshot if you're technical enough to split up OS and application data volumes on your local drive can potentially help but you still have registry or other issues. The best way to handle…

> A system snapshot for a personal computer is potentially a really difficult thing to restore (ex. you break the device and need a new one, unless it matches exactly restoring snapshot likely puts you in OS recovery mode). While I am sure the process has limitations, I think that modern Windows has some amount of flexibility there, because due to a defect I've recently had to swap out my mainboard + CPU, and the sys…

Yeah, plug and play drivers are much better today than years ago, they're also still challenging in many situations.

For someone with your technical skill level it's very manageable, most folks don't know what a mainboard is.

To an average user it's much easier to work with restoring their My Documents (and if they need full system state have Parallels save VMs under that folder path).

Post reply on HN