Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: NATS 2.12.1

jepsen.io

Post reply on HN
Print view View original

111–120 of 176 posts

Re: Jepsen: NATS 2.12.1

#111

Post by wseqyrku »

rdtsc wrote: post #14

> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…

> NATS only flushes data to disk every two minutes, but acknowledges operations immediately.

Wait, isn't that the whole point of acknowledgments? This is not acknowledgment, it's I'm a teapot.

Re: Jepsen: NATS 2.12.1

#112

Post by johncolanduoni »

colechristensen wrote: post #86
MrDarcy wrote: post #82

Earlier quoted context omitted.

The ivory tower standing in the way of delivering value I think.

To be more specific, goals of perfection where perfection does not at all matter.

What does bothering to read some distributed systems literature have to do with demanding unnecessary perfection? Did NATS have in their docs that JetStream accepted split brain conditions as a reality, or that metadata corruption could silently delete a topic? You could maybe argue the fsync default was a tradeoff, though I think it’s a bad one (not the existence of the flag, just the default being “false”). The rest are not the kind of bugs you expect to see in a 5 year old persistence layer.

Re: Jepsen: NATS 2.12.1

#114

Post by stmw »

johncolanduoni wrote: post #112
colechristensen wrote: post #86

Earlier quoted context omitted.

To be more specific, goals of perfection where perfection does not at all matter.

What does bothering to read some distributed systems literature have to do with demanding unnecessary perfection? Did NATS have in their docs that JetStream accepted split brain conditions as a reality, or that metadata corruption could silently delete a topic? You could maybe argue the fsync default was a tradeoff, though I think it’s a bad one (not the existence of the flag, just the default being “false”). The res…

Exactly, "losing data from acknowledged writes" is not failing to be perfect, it's failing to deliver on the (advertised) basics of storing your data.

Re: Jepsen: NATS 2.12.1

#115

Post by hxtk »

Thaxll wrote: post #39
rdtsc wrote: post #14

> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…

I don't think there is a modern database that have the safest options all turned on by default. For instance the default transaction model for PG is read commited not serializable One of the most used DB in the world is Redis, and by default they fsync every seconds not every operations.

CockroachDB is serializable by default, but I don’t know about their other settings.

Re: Jepsen: NATS 2.12.1

#116

Post by hxtk »

zbentley wrote: post #44
0xbadcafebee wrote: post #24

Earlier quoted context omitted.

Not flushing on every write is a very common tradeoff of speed over durability. Filesystems, databases, all kinds of systems do this. They have some hacks to prevent it from corrupting the entire dataset, but lost writes are accepted. You can often prevent this by enabling an option or tuning a parameter. > I wouldn't trust a product that doesn't default to safest options This would make most products suck, and requi…

I think “most people will have to turn on the setting to make things fast at the expense of durability” is a dubious assertion (plenty of system, even high-criticality ones, do not have a very high data rate and thus would not necessarily suffer unduly from e.g. fsync-every-write). Even if most users do turn out to want “fast_and_dangerous = true”, that’s not a particularly onerous burden to place on users: flip one…

I always think about the way you discover the problem. I used to say the same about RNG: if you need fast PRNG and you pick CSPRNG, you’ll find out when you profile your application because it isn’t fast enough. In the reverse case, you’ll find out when someone successfully guesses your private key.

If you need performance and you pick data integrity, you find out when your latency gets too high. In the reverse case, you find out when a customer asks where all their data went.

Re: Jepsen: NATS 2.12.1

#117

Post by akshayshah »

sreekanth850 wrote: post #113

this is absolutely shocking!Does kafka do fsync on every write?

No. Redpanda has made a lot of noise about this over the years [0], and Confluent's Jack Vanlightly has responded in a fair bit of detail [1].

[0]: https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s...

[1]: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...

Re: Jepsen: NATS 2.12.1

#118

Post by akshayshah »

lubesGordi wrote: post #50
rdtsc wrote: post #14

> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…

I don't know about Jetstream, but redis cluster would only ack writes after replicating to a majority of nodes. I think there is some config on standalone redis too where you can ack after fsync (which apparently still doesn't guarantee anything because of buffering in the OS). In any case, understanding what the ack implies is important, and I'd be frustrated if jetstream docs were not clear on that.

At least per the Redis docs, clusters acknowledge writes before they're replicated: https://redis.io/docs/latest/operate/oss_and_stack/managemen...

The docs explicitly state that clusters do not provide strong consistency and can lose acknowledged data.

Re: Jepsen: NATS 2.12.1

#119

Post by derefr »

otterley wrote: post #99
hbbio wrote: post #94

Earlier quoted context omitted.

> Blockchains are still a scam Did you actually look at the blockchain nodes implementation as of 2025 and what's in the roadmap? Ethereum nodes/L2s with optimistic or zk-proofs are probably the most advanced distributed databases that actually work. (not talking about "coins" and stuff obviously, another debate)

> Ethereum nodes/L2s with optimistic or zk-proofs are probably the most advanced distributed databases that actually work. What are you comparing against? Aren't they slower, less convenient, and less available than, say, DynamoDB or Spanner, both of which have been in full-service, reliable operation since 2012?

I think they mean big-D "Distributed", i.e. in the sense that a DHT is Distributed. Decentralized in both a logical and political sense.

A big DynamoDB/Spanner deployment is great while you can guarantee some benevolent (or just not-malevolent) org around to host the deployment for everyone else. But technologies of this type do not have any answer for the key problem of "ensure the infra survives its own founding/maintaining org being co-opted + enshittified by parties hostile to the central purpose of the network."

Blockchains — and all the overhead and pain that comes with them — are basically what you get when you take the classical small-D distributed database design, and add the components necessary to get that extra property.

Re: Jepsen: NATS 2.12.1

#120

Post by sreekanth850 »

akshayshah wrote: post #117
sreekanth850 wrote: post #113

this is absolutely shocking!Does kafka do fsync on every write?

No. Redpanda has made a lot of noise about this over the years [0], and Confluent's Jack Vanlightly has responded in a fair bit of detail [1]. [0]: https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s... [1]: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...

I think all modern system even scylla db do commit batch no fsync on every write, you either need throughput or durability both cannot exist together. Only thing what redpanda claim is you have to do replication before fsync so your data is not lost if the written node is dead due to a power failure. this is how scylla and cassandra works, if iam not wrong, so even if a node dead before the batch fsync, replication will be done before fsync from memtable,so other nodes will bring the durability and data loss is no longer true in a replicated setup. single node? obviously 100% data loss. but this is the trade off for a high tps system vs durable single ndoe system brings. its how you want to operate.
Post reply on HN