Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: NATS 2.12.1

jepsen.io

Post reply on HN
Print view View original

61–70 of 176 posts

Re: Jepsen: NATS 2.12.1

#61

Post by njuw »

mring33621 wrote: post #26
belter wrote: post #10

Earlier quoted context omitted.

[flagged]

NATS was originally made for simple, fast, ephemeral messaging. The persistence stuff is kinda new and it's not a surprise that there are limitations and bugs. You should see this report as a good thing, as it will add pressure for improvements.

> The persistence stuff is kinda new and it's not a surprise that there are limitations and bugs.

It's not really that new. The precursor to JetStream was NATS Streaming Server [1], which was first tagged almost 10 years ago [2].

[1] https://github.com/nats-io/nats-streaming-server

[2] https://github.com/nats-io/nats-streaming-server/releases/ta...

Re: Jepsen: NATS 2.12.1

#62

Post by awesome_dude »

stmw wrote: post #31

Every time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.

/me strokes my long grey beard and nods

People always think "theory is overrated" or "hacking is better than having a school education"

And then proceed to shoot themselves in the foot with "workarounds" that break well known, well documented, well traversed problem spaces

Re: Jepsen: NATS 2.12.1

#63

Post by KaiserPro »

billywhizz wrote: post #55
KaiserPro wrote: post #47

Earlier quoted context omitted.

oh sorry I was talking about NATS core. not jetstream. I'd be pretty sceptical about persistence

the OP was specifically about jetstream so i guess you just didn't read it?

just imagine I'm claude,

smoke bomb

Re: Jepsen: NATS 2.12.1

#65

Post by johncolanduoni »

mrkeen wrote: post #40
merb wrote: post #5

> 3.4 Lazy fsync by Default Why? Why do some databases do that? To have better performance in benchmarks? It’s not like that it’s ok to do that if you have a better default or at least write a lot about it. But especially when you run stuff in a small cluster you get bitten by stuff like that.

One of the perks of being distributed, I guess. The kind of failure that a system can tolerate with strict fsync but can't tolerate with lazy fsync (i.e. the software 'confirms' a write to its caller but then crashes) is probably not the kind of failure you'd expect to encounter on a majority of your nodes all at the same time.

It is if they’re in the same physical datacenter. Usually the way this is done is to wait for at least M replicas to fsync, but only require the data to be in memory for the rest. It smooths out the tail latencies, which are quite high for SSDs.

Re: Jepsen: NATS 2.12.1

#66

Post by to11mtm »

aphyr wrote: post #60
to11mtm wrote: post #59

Earlier quoted context omitted.

> This is how databases like Postgres work by default--see the `commit_delay` option here: https://www.postgresql.org/docs/8.1/runtime-config-wal.html I must note that the default for Postgres is that there is NO delay, which is a sane default. > You can batch up multiple operations into a single call to fsync. Ive done this in various messaging implementations for throughput, and it's actually fairly easy to do in m…

Ah, pardon me, spoke too quickly! I remembered that it fsynced by default, and offered batching, and forgot that the batch size is 0 by default. My bad!

Well the write is still tunable so you are still correct.

Just wanted to clarify that the default is still at least safe in case people perusing this for things to worry about, well, were thinking about worrying.

Love all of your work and writings, thank you for all you do!

Re: Jepsen: NATS 2.12.1

#68

Post by johncolanduoni »

hurturue wrote: post #11
belter wrote: post #10

Earlier quoted context omitted.

[flagged]

do you have a better solution? as they would say, NATS is a terrible message bus system, but all the others are worse

Pulsar can do most of what NATS can, but at a much higher cost in both compute and operations (though I haven’t seen a head-to-head of each with durability turned on), along with some simply different characteristics (like NATS being suitable for sidecar deployment). NATS is fantastic for ephemeral messaging, but some of this report is really concerning when JetStream has been shipping for years.

Re: Jepsen: NATS 2.12.1

#69

Post by ViewTrick1002 »

dzonga wrote: post #48

nats jetstream vs say redis streams - which one have people found easier to work with ?

When I worked with bounded Redis streams a couple of years ago we had to implement our own backpressure mechanism which was quite tricky to get right.

To implement backpressure without relying on out of band signals (distributed systems beware) you need to have a deep understanding of the entire redis streams architecture and how the the pending entries list, consumers groups, consumers etc. works and interacts to not lose data by overwriting yourself.

Unbounded would have been fine if we could spill to disk and periodically clean up the data, but this is redis.

Not sure if that has improved.

Post reply on HN