31–40 of 47 posts
Protocol Aware Recovery is really needed if you want Raft to tolerate disk corruption. https://www.usenix.org/conference/fast18/presentation/alagap...
I believe OP does not make any claims about arbitrary log corruption. Neither raft nor Kafka protocol can handle it. It is about losing tail of the log due to fsync failures or rather lack of fsyncs.
Earlier quoted context omitted.
" the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer" Well, kind of. fsync()'s job is only to ensure caches are written to durable storage. Its job is not to ensure the integrity of durable storage. Your idea of fsync()'s "Right Thing" is not quite correct, because these types of durable storage failures can happen outside of the context of a write -- so it's a…
> Again, fsync()'s job is only to ensure the data is moved to durable storage. The problem is that every layer in the stack lies. They say "yes it is definitely written to durable storage" when it is just in some cache layer and about to be written.
Not enterprise gear. Reliable storage does exist. I have tested many vendors myself, and gone through spec sheets under NDA (as mentioned above).
"They say "yes it is definitely written to durable storage" when it is just in some cache layer and about to be written."
Enterprise hardware contains batteries specifically designed so that caches can still be written out to durable storage in the event of power loss. Have you ever dealt with managing battery learning cycles on a Dell PERC?
Earlier quoted context omitted.
it tends to be true. we do things at the application tier to minimize these things. for example, assume you have to write A, B, C ... in syscalls it would be write( A ), flush() write( B ), flush() write( C ), flush() so if you add a debounce of say 4ms you get Write( A ), Write( B ), Write( C ), Flush() and saved 2 flush()-es This is common with protocol aware storage applications.
And I guess the idea is furthermore that just because you don't fsync after every write does not mean that you haven't fsync-ed before responding to the user request saying the data is stored durably. I assume that you do actually guarantee the flush before returning a success to the user.
Earlier quoted context omitted.
There is no way to know for sure that a Byzantine system is actually operating in a way that replicated copies of your data are actually safely written. Both due to the exact same issue of the drives themselves, and also that Byzantine system software is liable to have a variety of bugs and invalid states that will keep it operating as normal, even though the nodes are actually in a fault mode. (the problem is twofol…
> There is no way to know for sure that a Byzantine system is actually operating in a way that replicated copies of your data are actually safely written. Isn't this how Bitcoin adds to the ledger though? Using a merkle tree and slowing things down significantly with those 6+ confirmations.
Earlier quoted context omitted.
> There is no way to know for sure that a Byzantine system is actually operating in a way that replicated copies of your data are actually safely written. Isn't this how Bitcoin adds to the ledger though? Using a merkle tree and slowing things down significantly with those 6+ confirmations.
Technically Bitcoin can work without ever safely writing to disk. It's potentially all magical caches until the firmware decides to put bits to media.
Earlier quoted context omitted.
" the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer" Well, kind of. fsync()'s job is only to ensure caches are written to durable storage. Its job is not to ensure the integrity of durable storage. Your idea of fsync()'s "Right Thing" is not quite correct, because these types of durable storage failures can happen outside of the context of a write -- so it's a…
> Again, fsync()'s job is only to ensure the data is moved to durable storage. The problem is that every layer in the stack lies. They say "yes it is definitely written to durable storage" when it is just in some cache layer and about to be written.
I'm confused here. If I'm reading this correctly, the following is happening: - Node 3 is the leader. - Node 1 is isolated (failure #1). - 10 records are written. They are successful because a majority is still alive (persisted to Node 2 and 3). - Node 3 loses data (failure #2). - Node 1 comes back up again. However, isn't this two failures at the same time? Kafka with three nodes can only guarantee a single failure,…
Strongly consistent protocols such as a Paxos and Raft always choose consistency over availability and when consistency isn't certain they refuse to answer. Raft & Paxos: any number of nodes may be down, as soon as the majority is available a replicated system is available and doesn't lie. Kafka as it's described in the post( ): any number of nodes may be down, at most one power outage is allowed (loss of unsynced da…
https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-... talks about the case of a single failure and it shows how (a) Raft without fsync() loses ACK-ed messages and (b) Kafka without fsync() handles it fine.
This post on the other hand talks about a case where we have (a) one node being network partitioned, (b) the leader crashing, losing data, and combing back up again, all while (c) ZooKeeper doesn't catch that the leader crashed and elects another leader.
I think definitely the title/blurb should be updated to clarify that this is only in the "exceptional" case of >f failures.
I mean, the following paragraph seems completely misleading:
> Even the loss of power on a single node, resulting in local data loss of unsynchronized data, can lead to silent global data loss in a replicated system that does not use fsync, regardless of the replication protocol in use.
The next section (and the Kafka example) is talking about loss of power on a single node combined with another node being isolated. That's very different from just "loss of power on a single node".
Earlier quoted context omitted.
Strongly consistent protocols such as a Paxos and Raft always choose consistency over availability and when consistency isn't certain they refuse to answer. Raft & Paxos: any number of nodes may be down, as soon as the majority is available a replicated system is available and doesn't lie. Kafka as it's described in the post( ): any number of nodes may be down, at most one power outage is allowed (loss of unsynced da…
It just feels like two widely different scenarios we're talking about here. https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-... talks about the case of a single failure and it shows how (a) Raft without fsync() loses ACK-ed messages and (b) Kafka without fsync() handles it fine. This post on the other hand talks about a case where we have (a) one node being network partitioned, (b) the leader crashing, lo…
Earlier quoted context omitted.
Strongly consistent protocols such as a Paxos and Raft always choose consistency over availability and when consistency isn't certain they refuse to answer. Raft & Paxos: any number of nodes may be down, as soon as the majority is available a replicated system is available and doesn't lie. Kafka as it's described in the post( ): any number of nodes may be down, at most one power outage is allowed (loss of unsynced da…
It just feels like two widely different scenarios we're talking about here. https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-... talks about the case of a single failure and it shows how (a) Raft without fsync() loses ACK-ed messages and (b) Kafka without fsync() handles it fine. This post on the other hand talks about a case where we have (a) one node being network partitioned, (b) the leader crashing, lo…
I'm pretty sure Redpanda has identified a real issue here, but my only response to any situation where you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!" Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much…
Right. But that line of thinking gets you to a place where one is like “how does anything work” haha. In general this was a response to Confluent attempting to dismiss fsync() as a neat trick rather than an actual safety problem and why when we benchmarked we showcase the numbers we did.
https://jack-vanlightly.com/blog/2023/5/15/kafka-vs-redpanda...