161–170 of 176 posts
Earlier quoted context omitted.
and your point is ...?
I don't think they were making a point. Someone suggested using an LLM for this, someone then responded by using an LLM for it. What you draw from that seems entirely up to you. They don't seem to be making any claims or implying anything by doing so, just showing the result.
Earlier quoted context omitted.
You can have DeepWiki literally scan the source code and tell you: > 2. Delayed Sync Mode (Default) > In the default mode, writes are batched and marked with needSync = true for later synchronization filestore.go:7093-7097 . The actual sync happens during the next syncBlocks() execution. However, if you read DeepWiki's conclusion, it is far more optimistic than what Aphyr uncovered in real-world testing. > Durability…
> if you read DeepWiki's conclusion, it is far more optimistic Well, its an LLM ... of course its going to be optimistic. ;-)
Earlier quoted context omitted.
People overly beholden to tried and true 'known' way of addressing a problem space and not considering/belittling alternatives. Many of the things that have been most aggressively 'bitter lesson'ed in the last decade fall into this category.
Like this bug report? The things that have been "disrupted" haven't delivered - Blockchains are still a scam, Food delivery services are worse than before (Restaurants are worse off, the people making the deliveries are worse off), Taxis still needed to go back and vet drivers to ensure that they weren't fiends.
Every time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.
Earlier quoted context omitted.
It's not just better performance on latency benchmarks, it likely improves throughput as well because the writes will be batched together. Many applications do not require true durability and it is likely that many applications benefit from lazy fsync. Whether it should be the default is a lot more questionable though.
> Many applications do not require true durability Pretty much no application requires true durability.
When your system doesn't do things like fsync, you can't do that at all. X is 1. That is not what people expect.
Most people probably don't require X == Y, but they may have requirements that X > 1.
> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go…
> NATS only flushes data to disk every two minutes, but acknowledges operations immediately. Wait, isn't that the whole point of acknowledgments? This is not acknowledgment, it's I'm a teapot.
Earlier quoted context omitted.
I don't have a "school education" and I know plenty of theory, I certainly have read the papers cited in this test.
You might not have a school education, but you have educated yourself. It is unfortunately common to hear people complain that the theory one learns in school (or by determined self-study) is useless, which I think is what the geybeard comment you replied to intends to say.
1. School based is supposed to cover all the basics, self directed you have to know what the basics are, or find out, and then cover them.
2. School based study the teachers/lecturers are supposed to have checked all the available text on the subject and then share the best with the students (the teachers are the ones that ensure nobody goes down unproductive rabbitholes)
3. People can see from the qualifications that a person has met a certain standard, understands the subject, has got the knowledge, and can communicate that to a proscribed level.
Personal note, I have done both in different careers, and being "self taught" I realised that whilst I definitely knew more about one topic in the field than qualified individuals, I never knew what the complete set of study for the field was (i never knew how much they really knew, so could never fill the gaps I had)
In CS I gained my qualification in 2010, when i went to find work a lot of places were placing emphasis on self taught people who were deemed to be more creative, or more motivated, etc. When I did work with these individuals, without fail they were missing basic understanding of fundamentals, like data structures, well known algorithms, and so on.
Earlier quoted context omitted.
I always wondered why the fsync has to be lazy. It seems like the fsync's can be bundled up together, and the notification messages held for a few millis while the write completes. Similar to TCP corking. There doesn't need to be one fsync per consensus.
In practice, there must be a delay (from batching) if you fsync every transaction before acknowledging commit. The database would be unusably slow otherwise.
Every time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.
The only post in this thread that actually summarized the core findings of the study, namely: - ACKed messages can be silently lost due to minority-node corruption. - A single-bit corruption can cause some replicas to lose up to 78% of stored messages - Snapshot corruption can propagate and lead to entire stream deletion across the cluster. - The default lazy-fsync mode can drop minutes of acknowledged writes on a cr…
I have to note the following as a NATS fan:
- I am horrified at Jespen's reliability findings, however they do vindicate certain design decisions I made in the past
- 'Core NATS' is really mostly 'redis pubsub but better' and Core NATS is honestly awesome, low friction middleware. I've used it as part of eventing systems in the past and it works great.
- FWIW, There's an MQTT bridge that requires Jetstream, but if you're just doing QoS 0 you can work around the other warts.
- If you use Jetstream KV as a cache layer without real persistence (i.e. closer to how one uses Redis KV where it's just memory backed) you don't care about any of this. And again Jetstream KV IMO is better than Redis KV since they added TTL.
tl;dr - Jetstream's reliability is horrifying apparently but I stand by the statement that Core NATS and Ephermal KV is amazing.
Every time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.