Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Correct Backups Require Filesystem Snapshots

cyounkins.medium.com

Post reply on HN
Print view View original

61–70 of 78 posts

Re: Correct Backups Require Filesystem Snapshots

#61

Post by ek750 »

Havoc wrote: post #60
XorNot wrote: post #53

Earlier quoted context omitted.

This seems very unlikely without a lot of contributing factors. Unless you weren't running any RAID in which case ZFS didn't lose you data but it would've been quite upfront about corruption when it happened.

>This seems very unlikely That's what I thought given that it does have a good rep. Haven't given up on it yet, but definitely less trusting - 3 different drives in 3 different ways wasn't expected. That said - all consumer class gear in janky environments so there may very well be other factors lurking.

What was the layout of the ZFS pools? I've been using RAIDZ1 and RAIDZ2 for years now on ubuntu, and luckily have not detected any errors on my monthly scrubs. I did have bitrot on older drives previously when I just relied on EXT4, only to find out when trying to access an old file.

I'm using a Supermicro board, with ECC RAM, but anywhere from 4 to 6 drives in the pool (upgraded over 6 or 7 years now).

Re: Correct Backups Require Filesystem Snapshots

#62

Post by aidenn0 »

professoretc wrote: post #55
aidenn0 wrote: post #11

Earlier quoted context omitted.

It's been a while, but with e.g. Oracle 10, it was specifically stated in the docs that backing up the data-file(s), followed by the archive-logs, you would end up with a recoverable backup. I'm not sure if other databases are designed that way.

I've done it with SQL Server, but it wasn't automatic. I had to manually repair or remove the DB pages that were in use when the backup was made.

I think with Oracle you needed to replay the archive log; not sure if that was via rman or just a mount option.

Either way it was identical to crash recovery

Re: Correct Backups Require Filesystem Snapshots

#63

Post by drewzero1 »

michaelt wrote: post #4
drewzero1 wrote: post #3

Earlier quoted context omitted.

For a long time I didn't know where to start with backups, because I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.) Eventually I realized that some backups are better than no backups at all, and prioritized getting _something_ in place to at the very least have copies of important files copied regularly to another drive somewhere.

> I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.) It depends a lot on your use case. If you're backing up a production database system that's in constant use, then the files are almost certain to change while the backup is in progress. And a backup of everything on the database server except the data probably isn't what you want! On the other h…

In my case it was mostly office PCs that might have important documents or spreadsheets on them. I've been working on a redundant, networked backup solution to serve the whole office, but realized that I was building a henhouse while the chickens were already running loose. So I ordered some USB hard drives just to make sure things were getting copied.

Maintaining a super basic (but regularly verified) hourly incremental backup system has spared the people who use those PCs a lot of grief, plus saved me time trying to recover lost data.

Re: Correct Backups Require Filesystem Snapshots

#64

Post by a1369209993 »

liuliu wrote: post #5

Actually, I have a hard time to understand this. While snapshot reduces file corruptions, it is not a guarantee the best I understand. A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption). Thus, any discussion on file corruptions without a…

> While snapshot reduces file corruptions, it is not a guarantee the best I understand.

The idea is that you'll only encounter corruptions that the application could already hit due to crashes or power outages (and hence hopefully supports recovering from). For example, with naive reads, you might:

  read the first half of the file
  context switch to the application
  application writes to the first half of the file
  application fsyncs previous writes
  application writes to the second half of the file
  context switch back to you
  read the second half of the file
and end up with data in the second half of the file that the application normally only writes after it's sure that corresponding data has been written to the first half.

Re: Correct Backups Require Filesystem Snapshots

#65

Post by a1369209993 »

kazinator wrote: post #54
yakubin wrote: post #49

Earlier quoted context omitted.

You write "transaction", but Unix filesystems don't know anything about any transactions from the application's point of view. To them it's just a bunch of write(2) calls. How is a ZFS snapshot going to make sure that it snapshots the state either before or after what an application considers a "transaction", but implements with a sequence of write(2), fsync(2) etc. calls? (EDIT: The rest of this paragraph is rescind…

What ZFS will do it not capture a state before or after the transaction, but it will copy a state that the file actually has at some point during the transaction. The snapshot will have the state of the file which contains write_0, write_1, ...., write_k; and does not contain the effects of write_k+1, write_k+2, ... The serial copy of the file does not have that property. We may be able to find some k such that the b…

> assuming that the backup copy is made serially, and make sure that in every transaction, the writes occur in increasing offset order: the same direction. If the backup and transaction race in the same direction, then it's impossible for the backup to have "holes" whereby some older writes are missing.

Actually, that's the opposite of what you want: if the backup and application are racing in the same direction, then every write has a coinflip opportunity to either be included in the backup (if the application is slightly ahead and writes that block just before the backup reads it), or be ommited from the backup (if the backup is slightly ahead and reads that block just before the application writes it).

Whereas if the application writes from the end of the file to the start, the write where it meets the backup coming the opposite direction will be arbitrarily kept or lost, but in theory (but not in practice), any writes before it (toward the end of the file) will always be kept, and any writes after it (toward the start of the file) will always be lost, which is the same semantics you'd get from a snapshot at the time of that write.

Re: Correct Backups Require Filesystem Snapshots

#66

Post by kazinator »

a1369209993 wrote: post #65
kazinator wrote: post #54

Earlier quoted context omitted.

What ZFS will do it not capture a state before or after the transaction, but it will copy a state that the file actually has at some point during the transaction. The snapshot will have the state of the file which contains write_0, write_1, ...., write_k; and does not contain the effects of write_k+1, write_k+2, ... The serial copy of the file does not have that property. We may be able to find some k such that the b…

> assuming that the backup copy is made serially, and make sure that in every transaction, the writes occur in increasing offset order: the same direction. If the backup and transaction race in the same direction, then it's impossible for the backup to have "holes" whereby some older writes are missing. Actually, that's the opposite of what you want: if the backup and application are racing in the same direction, the…

You're right; racing in the same direction means there can be unwanted "holes": old blocks restored. E.g. database writes blocks 1 2 3 4; then the backup takes over and copies 1 2 3 4 5 6. Then database writes 5 6 7 8. Now 5 6 are new, and not backed up. Backup copies 7 8. Those are new and backed up. So 5 6 are a "stale hole" in the backup. We avoid this problem with opposite order; they can cross paths at most once.

Re: Correct Backups Require Filesystem Snapshots

#67

Post by iggldiggl »

bretpiatt wrote: post #47

A system snapshot for a personal computer is potentially a really difficult thing to restore (ex. you break the device and need a new one, unless it matches exactly restoring snapshot likely puts you in OS recovery mode). A volume snapshot if you're technical enough to split up OS and application data volumes on your local drive can potentially help but you still have registry or other issues. The best way to handle…

> A system snapshot for a personal computer is potentially a really difficult thing to restore (ex. you break the device and need a new one, unless it matches exactly restoring snapshot likely puts you in OS recovery mode).

While I am sure the process has limitations, I think that modern Windows has some amount of flexibility there, because due to a defect I've recently had to swap out my mainboard + CPU, and the system booted up just fine. The most major issue was that with the default drivers Windows had reverted to, the Ethernet controller didn't work, which could have been a little bit of a Catch-22 (no internet without the correct drivers, and no correct drivers without internet), but in practice I just downloaded the drivers on a different device and transferred them over via USB.

Re: Correct Backups Require Filesystem Snapshots

#68

Post by kaba0 »

ek750 wrote: post #61
Havoc wrote: post #60

Earlier quoted context omitted.

>This seems very unlikely That's what I thought given that it does have a good rep. Haven't given up on it yet, but definitely less trusting - 3 different drives in 3 different ways wasn't expected. That said - all consumer class gear in janky environments so there may very well be other factors lurking.

What was the layout of the ZFS pools? I've been using RAIDZ1 and RAIDZ2 for years now on ubuntu, and luckily have not detected any errors on my monthly scrubs. I did have bitrot on older drives previously when I just relied on EXT4, only to find out when trying to access an old file. I'm using a Supermicro board, with ECC RAM, but anywhere from 4 to 6 drives in the pool (upgraded over 6 or 7 years now).

To my knowledge ZFS indeed has a very very good track record, but Ubuntu did have an incorrect patch for it at a time, as I was enlightened by this comment in the thread: https://news.ycombinator.com/item?id=31292568

Perhaps ubuntu was the culprit, not ZFS?

Re: Correct Backups Require Filesystem Snapshots

#69

Post by kaba0 »

compsciphd wrote: post #41
jjnoakes wrote: post #31

Earlier quoted context omitted.

Would you mind elaborating? I'm not a Windows expert so I don't know off the top of my head what VSS provides that helps poorly written applications survive sudden power failure more robustly than without VSS in the picture, and how that's better than a file system point-in-time copy-on-write snapshot.

databases might need to do lots of writes. (example, could be modify multiple rows in a db). They can't neccessarily capture it into a single write operation. they will issue multiple. VSS is a method windows has to allow application to both be informed that a snapshot operation is in progress (and should quiesce their writes) as well as the ability to delay the snapshot operation from occurring (because they are in…

While the former maintainers post was very interesting, as I see it mostly helps with applications that already manage the sudden power loss use case. So a database can make use of that explicit API to somewhat better handle a snapshot, with no lingering transactions.

With only a COW filesystem snapshot, the end result will still be correct in case of programs that use transactions (as mentioned, dbs will either have commited all the transactions and the data is saved, or one was in-flight, those will be reverted and you are still in a consistent state). For programs that don’t support transactions both might be faulty, for those that support transactions COW will work correctly and for those that support this API COW will again work and might save a bit fresher version.

Re: Correct Backups Require Filesystem Snapshots

#70

Post by tatersolid »

jjnoakes wrote: post #42
compsciphd wrote: post #41

Earlier quoted context omitted.

databases might need to do lots of writes. (example, could be modify multiple rows in a db). They can't neccessarily capture it into a single write operation. they will issue multiple. VSS is a method windows has to allow application to both be informed that a snapshot operation is in progress (and should quiesce their writes) as well as the ability to delay the snapshot operation from occurring (because they are in…

Doesn't sound like that protects against power failure. Most databases I use keep a write-ahead log, which guards against power failure without requiring an external service. Multiple writes are handled without an issue. If power is lost, the database recovers automatically using whatever is on disk, no matter when it was interrupted. So a file system snapshot behaves the same and the database can start up from the s…

In many if not most production databases, that write-ahead log will be on a different volume or file system from the database files. This is often recommended in documentation for performance reasons.

So now it is quite possible you back up an older WAL coupled with a newer DB file that has partial writes inside. Boom.

This “multi-volume, multi-writer coordination” is the sort of problem VSS solves on windows. I’m not sure there’s anything equivalent on Linux or MacOS that can do multi-volume snapshots consistently.

Post reply on HN