Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we rolled our own consensus algorithm

quorumcontrol.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Why we rolled our own consensus algorithm

#31

Post by lifty »

zonotope wrote: post #17
sagichmal wrote: post #2

Not rolling your own consensus algorithm is approx. rules zero through ten of distributed systems programming. Nothing on this page provides any positive signal indicating unique (or even average) competence to the task. To wit, > We are borrowing all the best parts of existing technology and combining them in a unique and innovative way. To the best of my knowledge, none of the listed existing technologies have been…

We completely agree with you that, in general, you don't want to attempt to create an entirely new consensus mechanism. As you know, there are generally only 2 (perhaps 3) broad categories of byzantine tolerant consensus that boil down to PBFT and Nakamoto consensus. Ours is a flavor of PBFT designed to reduce message complexity. We also draw from Casper in order to get the incentives and game theory right which woul…

There is a new consensus algorithm called Avalanche, which apparently starts a new, 3rd category. You can read the white paper here: https://ipfs.io/ipfs/QmUy4jh5mGNZvLkjies1RWM4YuvJh5o2FYopNPV...

Re: Why we rolled our own consensus algorithm

#32

Post by tobowers »

lifty wrote: post #31
zonotope wrote: post #17

Earlier quoted context omitted.

We completely agree with you that, in general, you don't want to attempt to create an entirely new consensus mechanism. As you know, there are generally only 2 (perhaps 3) broad categories of byzantine tolerant consensus that boil down to PBFT and Nakamoto consensus. Ours is a flavor of PBFT designed to reduce message complexity. We also draw from Casper in order to get the incentives and game theory right which woul…

There is a new consensus algorithm called Avalanche, which apparently starts a new, 3rd category. You can read the white paper here: https://ipfs.io/ipfs/QmUy4jh5mGNZvLkjies1RWM4YuvJh5o2FYopNPV...

Absolutely. That’s the “perhaps 3” part of the above. Avalanche is really interesting. Can’t wait to see Ava.

Re: Why we rolled our own consensus algorithm

#33

Post by apersona »

eloff wrote: post #18
zzzcpan wrote: post #12

Earlier quoted context omitted.

If you have enough background in distributed systems to choose a consensus algorithm you are definitely capable of implementing one yourself and even invent your own. It's not a bad idea either, most existing consensus-based systems are absolute garbage. People just hugely overestimate how much those algorithms matter for production systems. EDIT: Note, I'm not talking about blockchains and byzantine fault tolerance,…

That's like saying if you have some crypto background you can implement your own crypto library. It's TERRIBLE advice. Read the Google paper on their chubby distributed locking system using paxos. They had a hard time getting it right - unless you think you're so much better than them. IIRC Leslie Lamport works (worked?) for them. Use an existing, well used, well scrutinized implementation of a known algorithm.

If you're a normal software engineer, I agree, you shouldn't implement it yourself.

But if you're a researcher/had a CS research background, I don't see why not. By definition research results in novel approaches (ideally better) than the current ones. You have to be willing to challenge the consensus.

PhD students who can prove that their algs work should work on it, like the results of this paper: https://raft.github.io

Re: Why we rolled our own consensus algorithm

#34

Post by eloff »

kjeetgill wrote: post #21
eloff wrote: post #18

Earlier quoted context omitted.

That's like saying if you have some crypto background you can implement your own crypto library. It's TERRIBLE advice. Read the Google paper on their chubby distributed locking system using paxos. They had a hard time getting it right - unless you think you're so much better than them. IIRC Leslie Lamport works (worked?) for them. Use an existing, well used, well scrutinized implementation of a known algorithm.

Are you sure? I thought Lamport has been at Microsoft Research since forever.

Ah the vagaries of the human memory.

Re: Why we rolled our own consensus algorithm

#35

Post by AgentME »

cdiddy2 wrote: post #25
davidgerard wrote: post #19

Earlier quoted context omitted.

Unless you're actually a computer scientist following the area ... all claims of solving the "blockchain trilemma" - decentralisation, security and efficiency - should be ignored unless and until they have an implementation to show you, and it survives the hostile Internet. Promises are cheap. e.g. IOHK is behind Cardano/ADA, which has a marvellous white paper making all sorts of mathematical promises ... and still r…

Luckily I am a computer scientist however it is extremely hard to keep up with everything going on in the blockchain space. There must be some validity behind the idea of weighted random turn-taking though since it is similar to how the next version(serenity) of ethereum will work based

I think the difference is how they choose whose turn it will be next. There's a big difference between a central authority picking people to take turns, and what Ethereum is working on.

Re: Why we rolled our own consensus algorithm

#36

Post by ggggtez »

zonotope wrote: post #24
newprint wrote: post #8

Any formal verification of you Algo ? Consensus algos are hard to implement correctly.

We're planning on formally verifying the algorithm once it stabilizes.

So... what was the purpose of me reading this non-announcement again?

Re: Why we rolled our own consensus algorithm

#38

Post by Geee »

lifty wrote: post #31
zonotope wrote: post #17

Earlier quoted context omitted.

We completely agree with you that, in general, you don't want to attempt to create an entirely new consensus mechanism. As you know, there are generally only 2 (perhaps 3) broad categories of byzantine tolerant consensus that boil down to PBFT and Nakamoto consensus. Ours is a flavor of PBFT designed to reduce message complexity. We also draw from Casper in order to get the incentives and game theory right which woul…

There is a new consensus algorithm called Avalanche, which apparently starts a new, 3rd category. You can read the white paper here: https://ipfs.io/ipfs/QmUy4jh5mGNZvLkjies1RWM4YuvJh5o2FYopNPV...

From a quick glance, isn't this somewhat similar to what Nano is doing? Nano does balance-weighed voting on conflicting blocks via gossip until the network arrives at consensus. https://nano.org/en/whitepaper

Re: Why we rolled our own consensus algorithm

#39

Post by dlubarov »

cremp wrote: post #11
eloff wrote: post #7

Earlier quoted context omitted.

This seems like a really, really bad idea. However, if everyone had that attitude we'd never improve on the existing art. So I'm glad some people are willing to risk it. But just to reiterate, if you're thinking of rolling your own consensus - or even implementing an existing consensus algorithm yourself (although that's much worse for Paxos than for Raft) - you're almost certainly making a big mistake.

Just to drive it home. Those who actually make their own encryption libraries know their stuff, and have been in the space for years. You should never, ever, roll your own encryption. If it's a requirement, then change your damn requirements, because just like consensus; it is not easy to get right, and the pros make mistakes.

I think it's worth distinguishing a few cases here:

1. Reimplementing an existing cryptographic primitive, such as SHA-256.

2. Inventing a new cryptographic primitive, with a reduction to existing cryptographic assumptions, and implementing it. A recent example is SPHINCS-256, which was proven secure in the random oracle model.

3. Postulating a new cryptographic hardness assumption, inventing a new cryptographic primitive based on it, and implementing it. Recent examples are IOTA's Curl hash function and StarkWare's Jarvis cipher.

#1 is risky, but at least the risk can be mitigated by having qualified peers review the implementation. #2 is riskier, but still, it can be mitigated by having qualified peers review the proof. #3 seems far more dangerous, since it involves conjecture.

What the Tupelo team is doing is like #2 -- risky, yes, but not comparable to #3.

Re: Why we rolled our own consensus algorithm

#40

Post by lou1306 »

therein wrote: post #29
kjeetgill wrote: post #21

Earlier quoted context omitted.

Are you sure? I thought Lamport has been at Microsoft Research since forever.

You're correct. Lamport has joined MSR in 2001 right after publishing "Time, Clocks, and the Ordering of Events in a Distributed System" and has been there ever since. I am not sure if he is still at the MV office but he was one of the reasons why MSR kept a presence in MV for a while.

Actually "Time, Clocks..." is a paper from the late 70s. Maybe you were thinking of the Paxos paper, "The Part-Time Parliament" (1998)?
Post reply on HN