Why `fsync()`: Losing unsynced data on a single node leads to global data loss

Post by Siecje » Tue, May 16, 2023, 5:18 PM UTC

throwaway09223 wrote: post #20

PreInternet01 wrote: post #2

I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…

" the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer" Well, kind of. fsync()'s job is only to ensure caches are written to durable storage. Its job is not to ensure the integrity of durable storage. Your idea of fsync()'s "Right Thing" is not quite correct, because these types of durable storage failures can happen outside of the context of a write -- so it's a…

Post by judofyr » Tue, May 16, 2023, 5:35 PM UTC

Post by eatonphil » Tue, May 16, 2023, 5:43 PM UTC

agallego wrote: post #24

bcrl wrote: post #19

Earlier quoted context omitted.

I stand by my position. The problem with fsync() is the question of when and where do errors get reported. If code checks errors on completion of each write it becomes significantly easier to know what write has failed. The error reporting is the granularity of the write being performed. In contrast, fsync() has to report errors for everything that has happened since the last fsync() call. If you are doing anything c…

it tends to be true. we do things at the application tier to minimize these things. for example, assume you have to write A, B, C ... in syscalls it would be write( A ), flush() write( B ), flush() write( C ), flush() so if you add a debounce of say 4ms you get Write( A ), Write( B ), Write( C ), Flush() and saved 2 flush()-es This is common with protocol aware storage applications.

Post by agallego » Tue, May 16, 2023, 6:10 PM UTC

eatonphil wrote: post #25

agallego wrote: post #24

Earlier quoted context omitted.

it tends to be true. we do things at the application tier to minimize these things. for example, assume you have to write A, B, C ... in syscalls it would be write( A ), flush() write( B ), flush() write( C ), flush() so if you add a debounce of say 4ms you get Write( A ), Write( B ), Write( C ), Flush() and saved 2 flush()-es This is common with protocol aware storage applications.

And I guess the idea is furthermore that just because you don't fsync after every write does not mean that you haven't fsync-ed before responding to the user request saying the data is stored durably. I assume that you do actually guarantee the flush before returning a success to the user.

Post by tzs » Tue, May 16, 2023, 6:21 PM UTC

PreInternet01 wrote: post #2

I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…