Why `fsync()`: Losing unsynced data on a single node leads to global data loss • Hacker News Board

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#41

Post by comet-engine » Tue, May 16, 2023, 10:09 PM UTC

judofyr wrote: post #37

rystsov wrote: post #30

Earlier quoted context omitted.

Strongly consistent protocols such as a Paxos and Raft always choose consistency over availability and when consistency isn't certain they refuse to answer. Raft & Paxos: any number of nodes may be down, as soon as the majority is available a replicated system is available and doesn't lie. Kafka as it's described in the post( ): any number of nodes may be down, at most one power outage is allowed (loss of unsynced da…

It just feels like two widely different scenarios we're talking about here. https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-... talks about the case of a single failure and it shows how (a) Raft without fsync() loses ACK-ed messages and (b) Kafka without fsync() handles it fine. This post on the other hand talks about a case where we have (a) one node being network partitioned, (b) the leader crashing, lo…

For what it’s worth, this form of loss wouldn’t be possible under KRaft since they (ironically?) use Raft for the metadata and elections. Ain’t nobody starting a new cluster with Zookeeper these days.

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#42

Post by bcrl » Tue, May 16, 2023, 11:24 PM UTC

agallego wrote: post #27

eatonphil wrote: post #25

Earlier quoted context omitted.

And I guess the idea is furthermore that just because you don't fsync after every write does not mean that you haven't fsync-ed before responding to the user request saying the data is stored durably. I assume that you do actually guarantee the flush before returning a success to the user.

exactly. you improve latency and throughput at the same time. is kinda cool. by delaying the reponses just a small bit you get huge benefits

Wrong. Use something like aio or io_uring to submit 4 asynchronous writes in a single system call and you'll get way better performance. The kernel has all kinds of infrastructure that tries to coalesce and batch things that the write()+fsync() syscalls make horribly inefficient, and in the modern world of really fast nvme drives, you want to make your calls into the device driver as efficient as possible. You'll burn far fewer CPU cycles by giving the kernel the whole set of i/os in one single go, burn less on synchronization. It really is better to avoid write() + fsync().

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#43

Post by rystsov » Wed, May 17, 2023, 12:05 AM UTC

judofyr wrote: post #37

rystsov wrote: post #30

Earlier quoted context omitted.

Strongly consistent protocols such as a Paxos and Raft always choose consistency over availability and when consistency isn't certain they refuse to answer. Raft & Paxos: any number of nodes may be down, as soon as the majority is available a replicated system is available and doesn't lie. Kafka as it's described in the post( ): any number of nodes may be down, at most one power outage is allowed (loss of unsynced da…

It just feels like two widely different scenarios we're talking about here. https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-... talks about the case of a single failure and it shows how (a) Raft without fsync() loses ACK-ed messages and (b) Kafka without fsync() handles it fine. This post on the other hand talks about a case where we have (a) one node being network partitioned, (b) the leader crashing, lo…

We can't ignore or pretend that network partitioning doesn't happen. When people talk about choosing two out of CAP the real question is C or A because P is out of our control.

When we combine network partitioning with single local data suffix loss it either leads to a consistency violation or to a system being unavailable desperate the majority of the nodes being are up. At the moment Kafka chooses availability over consistency.

Also I read Kafka source and the role of network partitioning doesn't seem to be crucial. I suspect that it's also possible to cause similar problem with a single node power-outage https://twitter.com/rystsov/status/1641166637356417027 and unfortunate timing

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#44

Post by rystsov » Wed, May 17, 2023, 1:11 AM UTC

ruuda wrote: post #18

> A system must use cutting-edge Byzantine fault-tolerant (BFT) replication protocols, which neither of these systems currently employ. Cutting-edge? pBFT (Practical Byzantine Fault Tolerance) was published in 1999. The first Tendermint release was in 2015. With few exceptions, almost all big proof of stake blockchains are powered by variations of pBFT and have been for many years.

Yep, I still consider them to be cutting edge. Paxos was written in 1990 but the industry adopted it only in 2010s. For example I've looked through pBFT and it doesn't mention reconfiguration protocol which is essential for industry use. I've found one from 2012 so it should be getting ripe by now.

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#45

Post by sean0- » Wed, May 17, 2023, 6:33 PM UTC

I wish articles such as this would also talk about the risk-adjusted cost of absolute data safety or provide treatment discussing the cost of ensuring the last 0.00001% of data is unconditionally guaranteed in the event of black swan events. For example, what's better for the business? Double EC2 operating costs from $100K/mo to $200K/mo or pay out $10K in credits/yr as a direct loss to the business plus whatever the reputational or regulatory cost? Sometimes you need 100%. Sometimes you don't. Not acknowledging this tradeoff makes the article very FUD'y and is a disservice to engineers who get sucked into black-and-white thinking.

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#46

Post by agallego » Fri, May 19, 2023, 10:40 PM UTC

bcrl wrote: post #42

agallego wrote: post #27

Earlier quoted context omitted.

exactly. you improve latency and throughput at the same time. is kinda cool. by delaying the reponses just a small bit you get huge benefits

Wrong. Use something like aio or io_uring to submit 4 asynchronous writes in a single system call and you'll get way better performance. The kernel has all kinds of infrastructure that tries to coalesce and batch things that the write()+fsync() syscalls make horribly inefficient, and in the modern world of really fast nvme drives, you want to make your calls into the device driver as efficient as possible. You'll bur…

what makes you think that we haven't tested this? seastar's io engine is defaulted to io-uring... there are about 10 things here to comment on. on optimized kernels we disable block coalescing at the kernel level, second we tell the kernel to use fifo, etc. these low hanging fruit was already something we've done for a very very long time.

Re: Why `fsync()`: Losing unsynced data on a single node leads to global data loss

#47

Post by agallego » Fri, May 19, 2023, 10:41 PM UTC

comet-engine wrote: post #40

agallego wrote: post #11

Earlier quoted context omitted.

Right. But that line of thinking gets you to a place where one is like “how does anything work” haha. In general this was a response to Confluent attempting to dismiss fsync() as a neat trick rather than an actual safety problem and why when we benchmarked we showcase the numbers we did.

Which numbers? The ones on the blog that shows Redpanda significantly lagging behind open source Kafka? https://jack-vanlightly.com/blog/2023/5/15/kafka-vs-redpanda...

this is a burner account created at the time this post was up.