If you are looking for a serverless alternative to JetStream, check out https://s2.dev Pros: unlimited streams with the durability of object storage – JetStream can only do a few K topics Cons: no consumer groups yet, it's on the agenda
[dead]
Jepsen: NATS 2.12.1
141–150 of 176 posts
Re: Jepsen: NATS 2.12.1
#142> 3.4 Lazy fsync by Default Why? Why do some databases do that? To have better performance in benchmarks? It’s not like that it’s ok to do that if you have a better default or at least write a lot about it. But especially when you run stuff in a small cluster you get bitten by stuff like that.
It's not just better performance on latency benchmarks, it likely improves throughput as well because the writes will be batched together. Many applications do not require true durability and it is likely that many applications benefit from lazy fsync. Whether it should be the default is a lot more questionable though.
I also think fsync before acking writes is a better default.
That aside, if you were to choose async for batching writes, their default value surprises me.
2 minutes seems like an eternity. Would you not get very good batching for throughout even at something like 2 seconds too?
Still not safe, but safer.
Re: Jepsen: NATS 2.12.1
#143nats jetstream vs say redis streams - which one have people found easier to work with ?
When I worked with bounded Redis streams a couple of years ago we had to implement our own backpressure mechanism which was quite tricky to get right. To implement backpressure without relying on out of band signals (distributed systems beware) you need to have a deep understanding of the entire redis streams architecture and how the the pending entries list, consumers groups, consumers etc. works and interacts to no…
I don't have a direct comment to add, but after working on the fringes of streams a bit, they've worked as advertised, but the API surface area for them is full of cases where, as you say, you have to kind of internalize the full architecture to really understand what's going on. It can be a bit overwhelming.
Re: Jepsen: NATS 2.12.1
#144Earlier quoted context omitted.
Have you tried running Jepsen against it?
We do deterministic simulation testing https://s2.dev/blog/dst https://s2.dev/blog/linearizability We have also adopted Antithesis for a more thorough DST environment, and plan to do more with it. One day we will engage Kyle to Jepsen, too. I'm not sure when though.
NATS claims to use Antithesis as well, so that's nothing comparatively speaking
Re: Jepsen: NATS 2.12.1
#145Every time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.
The only post in this thread that actually summarized the core findings of the study, namely:
- ACKed messages can be silently lost due to minority-node corruption.
- A single-bit corruption can cause some replicas to lose up to 78% of stored messages
- Snapshot corruption can propagate and lead to entire stream deletion across the cluster.
- The default lazy-fsync mode can drop minutes of acknowledged writes on a crash.
- A crash combined with network delay can cause persistent split-brain and divergent logs.
- Data loss even with “sync_interval = always” in presence of membership changes or partitions.
- Self-healing and replica convergence did not always work reliably after corruption.
…was not downvoted, but flagged... That is telling. Documented failure modes are apparently controversial. Also raises the question: What level of technical due diligence was performed by organizations like Mastercard, Volvo, PayPal, Baidu, Alibaba, or AT&T before adopting this system?
So what is next? Nominate NATS for the Silent Failure Peace Prize?
Re: Jepsen: NATS 2.12.1
#146Every time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.
You can have DeepWiki literally scan the source code and tell you: > 2. Delayed Sync Mode (Default) > In the default mode, writes are batched and marked with needSync = true for later synchronization filestore.go:7093-7097 . The actual sync happens during the next syncBlocks() execution. However, if you read DeepWiki's conclusion, it is far more optimistic than what Aphyr uncovered in real-world testing. > Durability…
and your point is ...?
Re: Jepsen: NATS 2.12.1
#147> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…
I don't think there is a modern database that have the safest options all turned on by default. For instance the default transaction model for PG is read commited not serializable One of the most used DB in the world is Redis, and by default they fsync every seconds not every operations.
FoundationDB provides strict serializability by default.
Re: Jepsen: NATS 2.12.1
#148For anyone dealing with databases, and especially distributed databases, I highly recommend reading the Jepsen page on consistency models: https://jepsen.io/consistency/models
It provides a dictionary of terms that we can use to have educated discussions, rather than throwing around terms like "ACID".
Re: Jepsen: NATS 2.12.1
#149> 3.4 Lazy fsync by Default Why? Why do some databases do that? To have better performance in benchmarks? It’s not like that it’s ok to do that if you have a better default or at least write a lot about it. But especially when you run stuff in a small cluster you get bitten by stuff like that.
It's not just better performance on latency benchmarks, it likely improves throughput as well because the writes will be batched together. Many applications do not require true durability and it is likely that many applications benefit from lazy fsync. Whether it should be the default is a lot more questionable though.
> Many applications do not require true durability
Pretty much no application requires true durability.
Re: Jepsen: NATS 2.12.1
#150Every time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.
The only post in this thread that actually summarized the core findings of the study, namely: - ACKed messages can be silently lost due to minority-node corruption. - A single-bit corruption can cause some replicas to lose up to 78% of stored messages - Snapshot corruption can propagate and lead to entire stream deletion across the cluster. - The default lazy-fsync mode can drop minutes of acknowledged writes on a cr…
> Nominate NATS for the Silent Failure Peace Prize?
One or two of the comments on GitHub by the NATS team in response to Issues opened by Kyle are also more than a bit cringeworthy.
Such as this one:
"Most of our production setups, and in fact Synadia Cloud as well is that each replica is in a separate AZ. These have separate power, networking etc. So the possibility of a loss here is extremely low in terms of due to power outages."
Which Kyle had to call them out on:
"Ah, I have some bad news here--placing nodes in separate AZs does not mean that NATS' strategy of not syncing things to disk is safe. See #7567 for an example of a single node failure causing data loss (and split-brain!)."
https://github.com/nats-io/nats-server/issues/7564#issuecomm...