Why `fsync()`: Losing unsynced data on a single node leads to global data loss
1–10 of 47 posts
Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss
#2I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!"
Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer without an NDA with the disk manufacturer and a hefty dose of proprietary IOCTLs (and even then, I would not bet my life on it).
In a typical 'Enterprise' stack, which would extend the above with a healthy dose of Fibre Channel over Ethernet, load-balancing switches, distributed RAM caches, and a bunch of other complexity, I would not even bet the life of my goldfish on it.
The article mentions Byzantine systems as a possible solution, but my only knowledge about those comes from reading https://www.usenix.org/system/files/login/articles/13_micken..., which makes me not exactly optimistic about their real-world applicability.
One thing that comes to mind when reading things like this is 'Distributed journaling for distributed systems', but since nobody seems to be doing that, it's probably a silly idea. I'll probably stick to SQLite write-ahead logs, at least I can wrap my mind around those...
Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss
#3I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…
Using fsync() to flush data to disk is, generally speaking, the wrong way to do things. fsync() is like using a piece of wood to hammer a nail into the ground. It'll work under very limited circumstances, but will fail horribly to do the heavy lifting required in other cases. fsync()'s semantics are complex and poorly defined, doubly so if multiple threads or processes are involved. Who gets the i/o error for a failed write? Nobody knows! fsync() is probably one of the worst APIs warts in Linux/Unix.
Developers competent in matters of storage generally use other methods / APIs like open()ing a file with O_SYNC and then checking the results of write() or pwrite() which will clearly identify when data is or is not successfully written to disk. If you want more performance, use async writes and it'll perform just fine. Seeing fsync() in code is a sign that the code has a bad smell and probably needs to be thoroughly reviewed.
Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss
#4I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…
> One thing that comes to mind when reading things like this is 'Distributed journaling for distributed systems', but since nobody seems to be doing that, it's probably a silly idea
That's exactly what Redpanda, BookKeeper, LogDevice, Pravega, and countless other proprietary systems at big tech companies do.
Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss
#5I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…
> Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer without an NDA with the disk manufacturer and a hefty dose of proprietary IOCTLs (and even then, I would not bet my life on it).
The answer is easy in all cases: No.
Media can always fail without notice.
Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss
#6I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…
Using fsync() to flush data to disk is, generally speaking, the wrong way to do things. fsync() is like using a piece of wood to hammer a nail into the ground. It'll work under very limited circumstances, but will fail horribly to do the heavy lifting required in other cases. fsync()'s semantics are complex and poorly defined, doubly so if multiple threads or processes are involved. Who gets the i/o error for a faile…
There may be performance reasons to still use fsync with O_DIRECT instead of only O_SYNC.
Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss
#7I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…
There is no way to know for sure that a Byzantine system is actually operating in a way that replicated copies of your data are actually safely written. Both due to the exact same issue of the drives themselves, and also that Byzantine system software is liable to have a variety of bugs and invalid states that will keep it operating as normal, even though the nodes are actually in a fault mode. (the problem is twofold: 1) the node and/or system not refusing writes when in a fault state, 2) the system not actually knowing that it's in a fault state) Even if you do all kinds of Jepsen simulation and mathematical proofs of the software (including the operating system!), you still can't trust the drives.
I think the only way to solve the problem is new storage firmware and hardware that is open and guarantees a write is done. I'm sure some companies may claim such functionality but we need an open source architecture and code to be sure.
In the meantime I think synchronous writes to multiple nodes is the safest option. Avoids complexity and bugs in fancy software, and the hardware is what it is.
Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss
#8Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss
#9I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…
Using fsync() to flush data to disk is, generally speaking, the wrong way to do things. fsync() is like using a piece of wood to hammer a nail into the ground. It'll work under very limited circumstances, but will fail horribly to do the heavy lifting required in other cases. fsync()'s semantics are complex and poorly defined, doubly so if multiple threads or processes are involved. Who gets the i/o error for a faile…
> sing fsync() to flush data to disk is, generally speaking, the wrong way to do things.
I think this is an old "statement" due to the Linux ext4 sync issue from quite awhile ago. As far as I know it has been fixed. I doubt it is discouraged on the BSDs.
But, curious, what is the Right Thing to do ? There are cases where you want to be sure the data reaches the platter.
On a mini I use to program on, if a filename started with an '@' sign, that said all writes went directly to disk. Based upon that were were able to create a roll-forward recovery for our custom applications. So, as far as I know, fsync(2) should accomplish the same thing using the open descriptor.
Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss
#10I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…
There is no way to know for sure that a Byzantine system is actually operating in a way that replicated copies of your data are actually safely written. Both due to the exact same issue of the drives themselves, and also that Byzantine system software is liable to have a variety of bugs and invalid states that will keep it operating as normal, even though the nodes are actually in a fault mode. (the problem is twofol…
> There is no way to know for sure that a Byzantine system is actually operating in a way that replicated copies of your data are actually safely written.
Isn't this how Bitcoin adds to the ledger though? Using a merkle tree and slowing things down significantly with those 6+ confirmations.