Earlier quoted context omitted.
/dev/null tends to lose a lot more data.
Just wait until the jepsen report on /dev/null. It's going to be brutal.
/dev/null works according to spec, can't accuse it of not doing something it has never promised
Jepsen: NATS 2.12.1
71–80 of 176 posts
Re: Jepsen: NATS 2.12.1
#72Every time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.
I've asked LLMs to do similar tasks and the results were very useful.
Re: Jepsen: NATS 2.12.1
#73> 3.4 Lazy fsync by Default Why? Why do some databases do that? To have better performance in benchmarks? It’s not like that it’s ok to do that if you have a better default or at least write a lot about it. But especially when you run stuff in a small cluster you get bitten by stuff like that.
It's not just better performance on latency benchmarks, it likely improves throughput as well because the writes will be batched together. Many applications do not require true durability and it is likely that many applications benefit from lazy fsync. Whether it should be the default is a lot more questionable though.
For transactional durability, the writes will definitely be batched ("group commit"), because otherwise throughput would collapse.
Re: Jepsen: NATS 2.12.1
#74Earlier quoted context omitted.
One of the perks of being distributed, I guess. The kind of failure that a system can tolerate with strict fsync but can't tolerate with lazy fsync (i.e. the software 'confirms' a write to its caller but then crashes) is probably not the kind of failure you'd expect to encounter on a majority of your nodes all at the same time.
It is if they’re in the same physical datacenter. Usually the way this is done is to wait for at least M replicas to fsync, but only require the data to be in memory for the rest. It smooths out the tail latencies, which are quite high for SSDs.
You can push the safety envelope a bit further and wait for your data to only be in memory in N separate fault domains. Yes, your favorite ultra-reliable cloud service may be doing this.
Re: Jepsen: NATS 2.12.1
#75Every time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.
/me strokes my long grey beard and nods People always think "theory is overrated" or "hacking is better than having a school education" And then proceed to shoot themselves in the foot with "workarounds" that break well known, well documented, well traversed problem spaces
certainly a narrative that is popular among the grey beard crowd, yes. in pretty much every field i've worked on, the opposite problem has been much much more common.
Re: Jepsen: NATS 2.12.1
#76> 3.4 Lazy fsync by Default Why? Why do some databases do that? To have better performance in benchmarks? It’s not like that it’s ok to do that if you have a better default or at least write a lot about it. But especially when you run stuff in a small cluster you get bitten by stuff like that.
I always wondered why the fsync has to be lazy. It seems like the fsync's can be bundled up together, and the notification messages held for a few millis while the write completes. Similar to TCP corking. There doesn't need to be one fsync per consensus.
In practice, there must be a delay (from batching) if you fsync every transaction before acknowledging commit. The database would be unusably slow otherwise.
Re: Jepsen: NATS 2.12.1
#77Earlier quoted context omitted.
It's not just better performance on latency benchmarks, it likely improves throughput as well because the writes will be batched together. Many applications do not require true durability and it is likely that many applications benefit from lazy fsync. Whether it should be the default is a lot more questionable though.
It’s like using a non-cryptographically secure RNG: if you don’t know enough to look for the fsync flag off yourself, it’s unlikely you know enough to evaluate the impact of durability on your application.
> if you don’t know enough to look for the fsync flag off yourself,
Yeah, it should use safe-defaults.
Then you can always go read the corners of the docs for the "go faster" mode.
Just like Postgres's infamous "non-durable settings" page... https://www.postgresql.org/docs/18/non-durability.html
Re: Jepsen: NATS 2.12.1
#78> 3.4 Lazy fsync by Default Why? Why do some databases do that? To have better performance in benchmarks? It’s not like that it’s ok to do that if you have a better default or at least write a lot about it. But especially when you run stuff in a small cluster you get bitten by stuff like that.
durability through replication and distribution and better throughput to build up more within the window on a lazy fsync
Re: Jepsen: NATS 2.12.1
#79> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…
> Well, if you don't fsync, you'll go fast, but you'll go even faster piping customer data to /dev/null, too. The trouble is that you need to specifically optimize for fsyncs, because usually it is either no brakes or hand-brake. The middle-ground of multi-transaction group-commit fsync seems to not exist anymore because of SSDs and massive IOPS you can pull off in general, but now it is about syscall context switche…
IOPS only solves throughput, not latency. You still need to saturate internal parallelism to get good throughput from SSDs, and that requires batching. Also, even double-digit microsecond write latency per transaction commit would limit you to only 10K TPS. It's just not feasible to issue individual synchronous writes for every transaction commit, even on NVMe.
tl;dr "multi-transaction group-commit fsync" is alive and well
Re: Jepsen: NATS 2.12.1
#80Earlier quoted context omitted.
/me strokes my long grey beard and nods People always think "theory is overrated" or "hacking is better than having a school education" And then proceed to shoot themselves in the foot with "workarounds" that break well known, well documented, well traversed problem spaces
certainly a narrative that is popular among the grey beard crowd, yes. in pretty much every field i've worked on, the opposite problem has been much much more common.
what's the opposite problem statement?