Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why `fsync()`: Losing unsynced data on a single node leads to global data loss

redpanda.com

Post reply on HN
Print view View original

11–20 of 47 posts

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#11

Post by agallego »

PreInternet01 wrote: post #2

I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…

Right. But that line of thinking gets you to a place where one is like “how does anything work” haha.

In general this was a response to Confluent attempting to dismiss fsync() as a neat trick rather than an actual safety problem and why when we benchmarked we showcase the numbers we did.

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#12

Post by agallego »

throwawaaarrgh wrote: post #7
PreInternet01 wrote: post #2

I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…

There is no way to know for sure that a Byzantine system is actually operating in a way that replicated copies of your data are actually safely written. Both due to the exact same issue of the drives themselves, and also that Byzantine system software is liable to have a variety of bugs and invalid states that will keep it operating as normal, even though the nodes are actually in a fault mode. (the problem is twofol…

That’s right. Raft uses sync writes which is what redpanda uses.

Complexity and industrial level reference impls are crucial

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#13

Post by insanitybit »

throwawaaarrgh wrote: post #7
PreInternet01 wrote: post #2

I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…

There is no way to know for sure that a Byzantine system is actually operating in a way that replicated copies of your data are actually safely written. Both due to the exact same issue of the drives themselves, and also that Byzantine system software is liable to have a variety of bugs and invalid states that will keep it operating as normal, even though the nodes are actually in a fault mode. (the problem is twofol…

> In the meantime I think synchronous writes to multiple nodes is the safest option.

This is what is being proposed though, right? Like no one is saying just use fsync, it's to use fsync across multiple systems, no?

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#15

Post by rdtsc »

PreInternet01 wrote: post #2

I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…

> you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!"

But it's safe to assume the right thing is not done if we haven't even called `fsync` to start with and just hope and pray that the dirty page flushing mechanism does it for us.

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#16

Post by deathanatos »

account42 wrote: post #5
PreInternet01 wrote: post #2

I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…

> Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer without an NDA with the disk manufacturer and a hefty dose of proprietary IOCTLs (and even then, I would not bet my life on it). The answer is easy in all cases: No. Media can always fail without notic…

Media can fail, but the parent there is saying they can fail with byzantine faults.

Most of us don't protect against such faults. (Neither Redpanda nor Kafka do, but that's not the point of the OP; the OP's point is that Kafka is not protecting against non-byzantine faults in the disk.)

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#17

Post by icedchai »

account42 wrote: post #5
PreInternet01 wrote: post #2

I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…

> Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer without an NDA with the disk manufacturer and a hefty dose of proprietary IOCTLs (and even then, I would not bet my life on it). The answer is easy in all cases: No. Media can always fail without notic…

Media failures after the data is written seems out of scope.

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#18

Post by ruuda »

> A system must use cutting-edge Byzantine fault-tolerant (BFT) replication protocols, which neither of these systems currently employ.

Cutting-edge? pBFT (Practical Byzantine Fault Tolerance) was published in 1999. The first Tendermint release was in 2015. With few exceptions, almost all big proof of stake blockchains are powered by variations of pBFT and have been for many years.

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#19

Post by bcrl »

jmclnx wrote: post #9
bcrl wrote: post #3

Earlier quoted context omitted.

Using fsync() to flush data to disk is, generally speaking, the wrong way to do things. fsync() is like using a piece of wood to hammer a nail into the ground. It'll work under very limited circumstances, but will fail horribly to do the heavy lifting required in other cases. fsync()'s semantics are complex and poorly defined, doubly so if multiple threads or processes are involved. Who gets the i/o error for a faile…

> sing fsync() to flush data to disk is, generally speaking, the wrong way to do things. I think this is an old "statement" due to the Linux ext4 sync issue from quite awhile ago. As far as I know it has been fixed. I doubt it is discouraged on the BSDs. But, curious, what is the Right Thing to do ? There are cases where you want to be sure the data reaches the platter. On a mini I use to program on, if a filename st…

I stand by my position. The problem with fsync() is the question of when and where do errors get reported. If code checks errors on completion of each write it becomes significantly easier to know what write has failed. The error reporting is the granularity of the write being performed.

In contrast, fsync() has to report errors for everything that has happened since the last fsync() call. If you are doing anything complicated like a database, which is virtually every single modern application, you need to have some idea of which writes have failed to figure out how to recover. fsync() gives you none of that. Different .

In kernel you get results for each i/o performed, so filesystems can make the appropriate decision about how to respond to a failure.

If you're building a high performance application, fsync() is a disaster. Unrelated writes will get bundled up when different threads call fsync() on the same file descriptor. Having multiple file descriptors open on the same file is silly and causes issues with applications that need to have many file descriptors open. The Linux kernel goes to great lengths to ensure that filesystems are high performance for writes on multiple threads.

fsync() sucks for anything more complex than "I have a single file to write out".

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#20

Post by throwaway09223 »

PreInternet01 wrote: post #2

I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…

" the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer"

Well, kind of. fsync()'s job is only to ensure caches are written to durable storage. Its job is not to ensure the integrity of durable storage. Your idea of fsync()'s "Right Thing" is not quite correct, because these types of durable storage failures can happen outside of the context of a write -- so it's a bit silly to point a finger at fsync().

For example, you can write bytes to disk today and the drive might experience corruption on that sector next week.

"(and even then, I would not bet my life on it)"

All hardware fails, so of course you shouldn't bet your life on it. That's why we have both onsite and offsite backups. The durable media itself can always fail and lose data -- even if there aren't any writes occurring. Data loss can even occur on a powered off system.

Again, fsync()'s job is only to ensure the data is moved to durable storage. Solving the problem of data loss is done in other ways -- for example raid5 addresses the problem of durable storage failing by using parity and extra copies.

The real takeaway from the blog is perhaps that Raft as a protocol is inadequate, in the same way a raid1 mirror system is inadequate in terms of protecting from data corruption on durable storage (which of the two mirrored drives has the correct version of the data?). I'm frankly surprised that this situation is undetectable. It's a solved problem in other storage layers.

"I'll probably stick to SQLite write-ahead logs, at least I can wrap my mind around those"

Well, keep in mind: those have the same problem in the event of durable storage failure.

Post reply on HN