Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: NATS 2.12.1

jepsen.io

Post reply on HN
Print view View original

41–50 of 176 posts

Re: Jepsen: NATS 2.12.1

#41

Post by aphyr »

millipede wrote: post #12
merb wrote: post #5

> 3.4 Lazy fsync by Default Why? Why do some databases do that? To have better performance in benchmarks? It’s not like that it’s ok to do that if you have a better default or at least write a lot about it. But especially when you run stuff in a small cluster you get bitten by stuff like that.

I always wondered why the fsync has to be lazy. It seems like the fsync's can be bundled up together, and the notification messages held for a few millis while the write completes. Similar to TCP corking. There doesn't need to be one fsync per consensus.

Yes, good call! You can batch up multiple operations into a single call to fsync. You can also tune the number of milliseconds or bytes you're willing to buffer before calling `fsync` to balance latency and throughput. This is how databases like Postgres work by default--see the `commit_delay` option here: https://www.postgresql.org/docs/8.1/runtime-config-wal.html

Re: Jepsen: NATS 2.12.1

#42

Post by RedShift1 »

KaiserPro wrote: post #38
petre wrote: post #32

Earlier quoted context omitted.

So is MQTT, why bother with NATS then?

MQTT doesn't have the same semantics. https://docs.nats.io/nats-concepts/core-nats/reqreply request reply is really useful if you need low latency, but reasonably efficient queuing. (making sure to mark your workers as busy when processing otherwise you get latency spikes. )

You can do request/reply with MQTT too, you just have to implement more bits yourself, whilst NATS has a nice API that abstracts that away for you.

Re: Jepsen: NATS 2.12.1

#43

Post by traceroute66 »

KaiserPro wrote: post #34
rdtsc wrote: post #14

> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…

NATS is very upfront in that the only thing that is guaranteed is the cluster being up. I like that, and it allows me to build things around it. For us when we used it back in 2018, it performed well and was easy to administer. The multi-language APIs were also good.

> NATS is very upfront in that the only thing that is guaranteed is the cluster being up.

Not so fast.

Their docs makes some pretty bold claims about JetStream....

They talk about JetStream addressing the "fragility" of other streaming technology.

And "This functionality enables a different quality of service for your NATS messages, and enables fault-tolerant and high-availability configurations."

And one of their big selling-points for JetStream is the whole "stora and replay" thing. Which implies the storage bit should be trustworthy, no ?

Re: Jepsen: NATS 2.12.1

#44

Post by zbentley »

0xbadcafebee wrote: post #24
rdtsc wrote: post #14

> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…

Not flushing on every write is a very common tradeoff of speed over durability. Filesystems, databases, all kinds of systems do this. They have some hacks to prevent it from corrupting the entire dataset, but lost writes are accepted. You can often prevent this by enabling an option or tuning a parameter. > I wouldn't trust a product that doesn't default to safest options This would make most products suck, and requi…

I think “most people will have to turn on the setting to make things fast at the expense of durability” is a dubious assertion (plenty of system, even high-criticality ones, do not have a very high data rate and thus would not necessarily suffer unduly from e.g. fsync-every-write).

Even if most users do turn out to want “fast_and_dangerous = true”, that’s not a particularly onerous burden to place on users: flip one setting, and hopefully learn from the setting name or the documentation consulted when learning about it that it poses operational risk.

Re: Jepsen: NATS 2.12.1

#45

Post by KaiserPro »

belter wrote: post #10
clemlesne wrote: post #6

NATS is a fantastic piece of software. But doc’s unpractical and half backed. That’s a shame to be required to retro engineer the software from GitHub to know the auth schemes.

[flagged]

NATS is ephemeral. if you can accept that, then you'll be fine.

Re: Jepsen: NATS 2.12.1

#46

Post by KaiserPro »

RedShift1 wrote: post #42
KaiserPro wrote: post #38

Earlier quoted context omitted.

MQTT doesn't have the same semantics. https://docs.nats.io/nats-concepts/core-nats/reqreply request reply is really useful if you need low latency, but reasonably efficient queuing. (making sure to mark your workers as busy when processing otherwise you get latency spikes. )

You can do request/reply with MQTT too, you just have to implement more bits yourself, whilst NATS has a nice API that abstracts that away for you.

oh indeed, and clusters nicely.

Re: Jepsen: NATS 2.12.1

#47

Post by KaiserPro »

traceroute66 wrote: post #43
KaiserPro wrote: post #34

Earlier quoted context omitted.

NATS is very upfront in that the only thing that is guaranteed is the cluster being up. I like that, and it allows me to build things around it. For us when we used it back in 2018, it performed well and was easy to administer. The multi-language APIs were also good.

> NATS is very upfront in that the only thing that is guaranteed is the cluster being up. Not so fast. Their docs makes some pretty bold claims about JetStream.... They talk about JetStream addressing the "fragility" of other streaming technology. And "This functionality enables a different quality of service for your NATS messages, and enables fault-tolerant and high-availability configurations." And one of their bi…

oh sorry I was talking about NATS core. not jetstream. I'd be pretty sceptical about persistence

Re: Jepsen: NATS 2.12.1

#49

Post by johncolanduoni »

aaronbwebber wrote: post #18
merb wrote: post #5

> 3.4 Lazy fsync by Default Why? Why do some databases do that? To have better performance in benchmarks? It’s not like that it’s ok to do that if you have a better default or at least write a lot about it. But especially when you run stuff in a small cluster you get bitten by stuff like that.

It's not just better performance on latency benchmarks, it likely improves throughput as well because the writes will be batched together. Many applications do not require true durability and it is likely that many applications benefit from lazy fsync. Whether it should be the default is a lot more questionable though.

It’s like using a non-cryptographically secure RNG: if you don’t know enough to look for the fsync flag off yourself, it’s unlikely you know enough to evaluate the impact of durability on your application.

Re: Jepsen: NATS 2.12.1

#50

Post by lubesGordi »

rdtsc wrote: post #14

> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…

I don't know about Jetstream, but redis cluster would only ack writes after replicating to a majority of nodes. I think there is some config on standalone redis too where you can ack after fsync (which apparently still doesn't guarantee anything because of buffering in the OS). In any case, understanding what the ack implies is important, and I'd be frustrated if jetstream docs were not clear on that.
Post reply on HN