Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: Radix DLT 1.0-Beta.35.1

jepsen.io

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Jepsen: Radix DLT 1.0-Beta.35.1

#51

Post by cmiles74 »

redwood wrote: post #32

I have a pretty much opposite perspective on Jepsen than most of the folks here. My feeling is that essentially no distributed system is perfect or without trade offs (and certainly no single node system is perfect and wothout tradeoffs) and Jepsen posts basically make that clear over and over again like some form of techie outrage porn... but the tone and implication is as if somehow there is some alternative panace…

I disagree that the analysis articles from Jepsen, taken as whole, are arguing for a "perfect distributed system without trade offs". In my experience these articles are fairly even-handed, indeed, many products make some changes based on the analysis and then contract for a follow-up. That the "tone and implication is as if somehow there is some alternative panacea without tradeoffs..." is not something I observed in this analysis or the previous ones that I have read.

In my opinion the Rethink DB 2.1.5 report went fairly well for that project. If the claims are aligned with the reality of the product, it's clear the Jepsen report will highlight that.

https://jepsen.io/analyses/rethinkdb-2-1-5

Re: Jepsen: Radix DLT 1.0-Beta.35.1

#52

Post by TheDong »

redwood wrote: post #32

I have a pretty much opposite perspective on Jepsen than most of the folks here. My feeling is that essentially no distributed system is perfect or without trade offs (and certainly no single node system is perfect and wothout tradeoffs) and Jepsen posts basically make that clear over and over again like some form of techie outrage porn... but the tone and implication is as if somehow there is some alternative panace…

If you want to look at a Jepsen test which shows a system mostly working as designed, the etcd 3.4.3 one (https://jepsen.io/analyses/etcd-3.4.3) is a much nicer read. Quoting from the discussion section, "etcd 3.4.3 lived up to its claims for key-value operations: we observed nothing but strict-serializable consistency for reads, writes, and even multi-key transactions, during process pauses, crashes, clock skew, network partitions, and membership changes"

That's what a successful test looks like. Sure, they found correctness issues with the lock api, but they were able to lend some confidence to etcd's core api. There might be bugs, but jepsen didn't observe any.

Sure, that's not proof that systems don't have tradeoffs, but it's proof that jepsen doesn't literally always find consistency issues or other core problems. Said another way, Jepsen is testing for correctness issues in systems. They either find some or they don't. These results are interesting, even if they are not a panacea.

There are other tradeoffs, but for most systems, correctness is important enough it merits consideration on its own right.

I don't think readers of jepsen misunderstand what's being tested or what it means, nor do they misunderstand that there are other tradeoffs to consider, so I think your comment is off the mark.

By analogy, if we were reading a post about "I load-tested this bridge which claims to support 50 tons of weight, and it broke at 5 tons", no one would be saying "yeah, but there are tradeoffs for bridges. This post just makes it clear that there's tradeoffs. If you make the bridge stronger, it would be more expensive, and to imply there's not tradeoffs is misleading. An astute reader might deduce that they should never drive on bridges again".

I don't think that would be a reasonable interpretation of such a post, nor do I think the interpretation you espouse here portrays an accurate sentiment.

Re: Jepsen: Radix DLT 1.0-Beta.35.1

#53

Post by [deleted] »

raphlinus wrote: post #33

It's funny how the comments here are polarized, some of them claiming that Jepsen slaughtered RDX, others that it proved that the consensus layer is rock solid. Let's appreciate this for what it is. Blockchains are, at their heart, a type of database (or at the very least a ledger which can be the foundation on which some subset of database semantics can be layered). Performance and reliability are empirical claims w…

[deleted]

Re: Jepsen: Radix DLT 1.0-Beta.35.1

#54

Post by TheDong »

eklavya wrote: post #40
redwood wrote: post #39

Earlier quoted context omitted.

But the contrived nature of Jepsen tests is so different than the real world. In the real world no system behaves exactly the same as it was designed to behave; the real world has cosmic rays, earthquakes and everything in between. So no statement about a software system can ever be simply accepted as a fundamental fact and so to prove that software systems break is misleading for what most users need to understand a…

We do know that some databases fare much better than others and that’s useful to many. In your analogy it would be many builders claiming their buildings to be earthquake proof with only some actually being it. Thanks to Jepsen, customers know, where to buy.

Note, this analogy is slightly off for using the phrase "some actually being it". jepsen cannot prove "earthquake proof" (correctness).

Rather, it would be "Builders claim their buildings are earthquake proof, and jepson was able to show a subset collapse in earthquakes. The rest may or may not be earthquake proof".

That's still very valuable. It's really valuable to know when something is wrong. It would be more valuable to know that something is definitely right (correct / "earthquake proof"), but jepsen cannot prove that.

Re: Jepsen: Radix DLT 1.0-Beta.35.1

#55

Post by aphyr »

redwood wrote: post #44
eklavya wrote: post #40

Earlier quoted context omitted.

We do know that some databases fare much better than others and that’s useful to many. In your analogy it would be many builders claiming their buildings to be earthquake proof with only some actually being it. Thanks to Jepsen, customers know, where to buy.

I'm genuinely curious to hear which database is the winner from your perspective based on Jepsen's insight?

If you find out, let me know! Folks are always asking me "What database is best?" and I'm just shuffling my feet and going "Uhhhh, well, systems are really hard to build correctly, there's lots of tradeoffs, you probably want something with a proven replication model under the hood but so much depends on workload and operational characteristics beyond just safety... If you find a database you love please tell me about it because I like hearing people's stories..."

FWIW, I think you're raising really good questions in this thread. Qualitative safety is highly contextual, depending on fault model, significance of individual operations, concurrency, throughput, latency demands, operational characteristics, data volume, etc etc., and I try to touch on that in the "Toward a Culture of Safety" section in this report. Hopefully that resonated for you.

Re: Jepsen: Radix DLT 1.0-Beta.35.1

#56

Post by stefan_ »

redwood wrote: post #39
belter wrote: post #38

Earlier quoted context omitted.

Although I agree with your point, on language and its ambiguity, I would argue that is a different claim of the one you made above, and that I replied to. When he demonstrated that Riak was dropping 30-70% of writes, even with the strongest consistency settings, or that Mongo had multiple scenarios of data loss, we are not talking about the subtleties of the English idiom

But the contrived nature of Jepsen tests is so different than the real world. In the real world no system behaves exactly the same as it was designed to behave; the real world has cosmic rays, earthquakes and everything in between. So no statement about a software system can ever be simply accepted as a fundamental fact and so to prove that software systems break is misleading for what most users need to understand a…

This is such a clown take. Read any root cause analysis of your favorite cloud operator and you will find all of the scenarios that Jepsen tests for in those. And you know what's even better? They can be solved!

Re: Jepsen: Radix DLT 1.0-Beta.35.1

#57

Post by [deleted] »

redwood wrote: post #44
eklavya wrote: post #40

Earlier quoted context omitted.

We do know that some databases fare much better than others and that’s useful to many. In your analogy it would be many builders claiming their buildings to be earthquake proof with only some actually being it. Thanks to Jepsen, customers know, where to buy.

I'm genuinely curious to hear which database is the winner from your perspective based on Jepsen's insight?

[deleted]

Re: Jepsen: Radix DLT 1.0-Beta.35.1

#58

Post by exdsq »

aphyr wrote: post #19
faraz85 wrote: post #15

Earlier quoted context omitted.

As I understand from the report, no request was made to remove the content but to leave terms like "liveness break" and "safety break" out of the abstract until those terms were defined in the main report.

You understand incorrectly.

Did you even read the report aphyr?

/s

Re: Jepsen: Radix DLT 1.0-Beta.35.1

#59

Post by exdsq »

Mleekko wrote: post #47
raphlinus wrote: post #43

Earlier quoted context omitted.

Makes you wonder why they ended the collaboration in November and didn't continue to retain him to validate that the new stuff lives up to its claims, doesn't it?

It's not a secret at all. The response from the Radix CEO: "We re-used the part of his testing harness, and then re-created the other critical tests to determine if the errors detected where still present. He is fantastically expensive and booked 3-6 months in advance on average. We'll definite re-deploy this kind of testing again, but we'll save it for another bigger release, rather than a patch where the identified…

I wonder how much Aphyr charges for a Jespon audit

Re: Jepsen: Radix DLT 1.0-Beta.35.1

#60

Post by serverholic »

raphlinus wrote: post #33

It's funny how the comments here are polarized, some of them claiming that Jepsen slaughtered RDX, others that it proved that the consensus layer is rock solid. Let's appreciate this for what it is. Blockchains are, at their heart, a type of database (or at the very least a ledger which can be the foundation on which some subset of database semantics can be layered). Performance and reliability are empirical claims w…

I would love for a highly reputable team or individual to test blockchains and their claims. However, the testing should be done in a way that respects the fact that blockchains are a different type of database and should be judged as such.

For example, because of the blockchain trilemma, increasing transaction speed isn't always a good thing since it could sacrifice decentralization.

Post reply on HN