Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: NATS 2.12.1

jepsen.io

Post reply on HN
Print view View original

121–130 of 176 posts

Re: Jepsen: NATS 2.12.1

#121

Post by [deleted] »

Thaxll wrote: post #39
rdtsc wrote: post #14

> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…

I don't think there is a modern database that have the safest options all turned on by default. For instance the default transaction model for PG is read commited not serializable One of the most used DB in the world is Redis, and by default they fsync every seconds not every operations.

[deleted]

Re: Jepsen: NATS 2.12.1

#122

Post by LaGrange »

colechristensen wrote: post #86
MrDarcy wrote: post #82

Earlier quoted context omitted.

The ivory tower standing in the way of delivering value I think.

To be more specific, goals of perfection where perfection does not at all matter.

Last time I was at school requirement analysis was a thing, but do go off.

Re: Jepsen: NATS 2.12.1

#123

Post by ADefenestrator »

maxmcd wrote: post #17

> > You can force an fsync after each messsage [sic] with always, this will slow down the throughput to a few hundred msg/s. Is the performance warning in the NATS possible to improve on? Couldn't you still run fsync on an interval and queue up a certain number of writes to be flushed at once? I could imagine latency suffering, but batches throughput could be preserved to some extent?

Yes, this is a reasonably common strategy. It's how Cassandra's batch and group commit modes work, and Postgres has a similar option. Hopefully NATS will implement something similar eventually.

Re: Jepsen: NATS 2.12.1

#124

Post by drdrey »

charcircuit wrote: post #110
drdrey wrote: post #100

Earlier quoted context omitted.

the big difference is the trust assumption, anyone can join or leave the network of nodes at any time

I think you are being downvoted because Ethereum requires you to stake 32 Eth (about $100k), and the entry queue right now is about 9 days and the exit queue is about 20 days. So only people with enough capital can join the network and it takes quite some time to join or leave as opposed to being able to do it at any time you want.

ok but these are details, the point is that the operators of the database are external, selfish and fluctuating

Re: Jepsen: NATS 2.12.1

#125

Post by andersmurphy »

mysfi wrote: post #85

Curious about the differences between content on aphyr.com/tags/jepsen and jepsen.io/analyses. I recently discovered aphyr.com and was excited about the potential insights!

Highly recommend you check out the interview series they are a lot of fun.

> They will refuse, of course, and ever so ashamed, cite a lack of culture fit. Alight upon your cloud-pine, and exit through the window. This place could never contain you.

https://aphyr.com/posts/340-reversing-the-technical-intervie...

Re: Jepsen: NATS 2.12.1

#126

Post by andersmurphy »

Thaxll wrote: post #39
rdtsc wrote: post #14

> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…

I don't think there is a modern database that have the safest options all turned on by default. For instance the default transaction model for PG is read commited not serializable One of the most used DB in the world is Redis, and by default they fsync every seconds not every operations.

SQLite is alway serializable and by default has synchronous=Full so fsync on every commit.

The problem is it has terrible defaults for performance (in the context of web servers). Like just bad options legacy options not ones that make it less robust. Ie cache size ridiculously small, temp tables not in memory, WAL off so no concurrent reads/writes etc.

Re: Jepsen: NATS 2.12.1

#127

Post by andersmurphy »

0xbadcafebee wrote: post #91
TheTaytay wrote: post #57

Earlier quoted context omitted.

> Filesystems, databases, all kinds of systems do this. They have some hacks to prevent it from corrupting the entire dataset, but lost writes are accepted. Woah, those are _really_ strong claims. "Lost writes are accepted"? Assuming we are talking about "acknowledged writes", which the article is discussing, I don't think it's true that this is a common default for databases and filesystems. Perhaps databases or K/V…

All filesystems that I'm aware of don't sync to disk on every write by default, and you absolutely can lose data. You have to intentionally enable sync. And even then the disk can still lose the writes. Most (all?) NoSQL solutions are also eventual-consistency by default which means they can lose data. That's how Mongo works. It syncs a journal every 30-100 ms, and it syncs full writes at a configurable delay. Mongo…

I'd argue with mongo a lot of people use it because it has fantastic marketing.

https://nemil.com/2017/08/29/the-marketing-behind-mongodb/

Re: Jepsen: NATS 2.12.1

#128

Post by PeterCorless »

stmw wrote: post #31

Every time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.

You can have DeepWiki literally scan the source code and tell you:

> 2. Delayed Sync Mode (Default)

> In the default mode, writes are batched and marked with needSync = true for later synchronization filestore.go:7093-7097 . The actual sync happens during the next syncBlocks() execution.

However, if you read DeepWiki's conclusion, it is far more optimistic than what Aphyr uncovered in real-world testing.

> Durability Guarantees

> Even with delayed fsyncs, NATS provides protection against data loss through:

> 1. Write-Ahead Logging: Messages are written to log files before being acknowledged

> 2. Periodic Sync: The sync timer ensures data is eventually flushed to disk

> 3. State Snapshots: Full state is periodically written to index.db files filestore.go:9834-9850

> 4. Error Handling: If sync operations fail, NATS attempts to rebuild state from existing data filestore.go:7066-7072"

https://deepwiki.com/search/will-nats-lose-uncommitted-wri_b...

Re: Jepsen: NATS 2.12.1

#129

Post by williamstein »

williamstein wrote: post #70

https://github.com/williamstein/nats-bugs

For example, https://github.com/williamstein/nats-bugs/issues/5 links to a discussion I have with them about data loss, where they fundamentally don't understand that their incorrect defaults lead to data loss on the application side. It's weird.

I got very deep into using NATS last year, and then realized the choices it makes for persistence are really surprising. Another horrible example if that server startup time is O(number of streams), with a big constant; this is extremely painful to hit in production.

I ended up implementing from scratch something with the same functionality (for me as NATS server + Jetstream), but based on socket.io and sqlite. It works vastly better for my use cases, since socketio and sqlite are so mature.

Post reply on HN