Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we rolled our own consensus algorithm

quorumcontrol.com

Post reply on HN
Print view View original

11–20 of 44 posts

Re: Why we rolled our own consensus algorithm

#11

Post by cremp »

eloff wrote: post #7
sagichmal wrote: post #2

Not rolling your own consensus algorithm is approx. rules zero through ten of distributed systems programming. Nothing on this page provides any positive signal indicating unique (or even average) competence to the task. To wit, > We are borrowing all the best parts of existing technology and combining them in a unique and innovative way. To the best of my knowledge, none of the listed existing technologies have been…

This seems like a really, really bad idea. However, if everyone had that attitude we'd never improve on the existing art. So I'm glad some people are willing to risk it. But just to reiterate, if you're thinking of rolling your own consensus - or even implementing an existing consensus algorithm yourself (although that's much worse for Paxos than for Raft) - you're almost certainly making a big mistake.

Just to drive it home.

Those who actually make their own encryption libraries know their stuff, and have been in the space for years.

You should never, ever, roll your own encryption.

If it's a requirement, then change your damn requirements, because just like consensus; it is not easy to get right, and the pros make mistakes.

Re: Why we rolled our own consensus algorithm

#12

Post by zzzcpan »

eloff wrote: post #7
sagichmal wrote: post #2

Not rolling your own consensus algorithm is approx. rules zero through ten of distributed systems programming. Nothing on this page provides any positive signal indicating unique (or even average) competence to the task. To wit, > We are borrowing all the best parts of existing technology and combining them in a unique and innovative way. To the best of my knowledge, none of the listed existing technologies have been…

This seems like a really, really bad idea. However, if everyone had that attitude we'd never improve on the existing art. So I'm glad some people are willing to risk it. But just to reiterate, if you're thinking of rolling your own consensus - or even implementing an existing consensus algorithm yourself (although that's much worse for Paxos than for Raft) - you're almost certainly making a big mistake.

If you have enough background in distributed systems to choose a consensus algorithm you are definitely capable of implementing one yourself and even invent your own. It's not a bad idea either, most existing consensus-based systems are absolute garbage. People just hugely overestimate how much those algorithms matter for production systems.

EDIT: Note, I'm not talking about blockchains and byzantine fault tolerance, but in context of Paxos/Raft.

Re: Why we rolled our own consensus algorithm

#14

Post by elfakyn »

This sounds like a terrible idea. Similarly to implementing your own encryption, having a custom consensus algorithm can create additional vulnerabilities -- a custom consensus algorithm isn't as tested and vetted as an existing, widely-used one.

In security, you generally don't want "unique and innovative", you want tested, verified, robust algorithms that have been gone over with a fine-toothed comb.

Re: Why we rolled our own consensus algorithm

#15

Post by zzzcpan »

cremp wrote: post #11
eloff wrote: post #7

Earlier quoted context omitted.

This seems like a really, really bad idea. However, if everyone had that attitude we'd never improve on the existing art. So I'm glad some people are willing to risk it. But just to reiterate, if you're thinking of rolling your own consensus - or even implementing an existing consensus algorithm yourself (although that's much worse for Paxos than for Raft) - you're almost certainly making a big mistake.

Just to drive it home. Those who actually make their own encryption libraries know their stuff, and have been in the space for years. You should never, ever, roll your own encryption. If it's a requirement, then change your damn requirements, because just like consensus; it is not easy to get right, and the pros make mistakes.

Consensus is not crypto, it's ridiculously trivial in comparison and usually so useless, that nobody even bothers doing it correctly all the way to the end user.

Re: Why we rolled our own consensus algorithm

#17

Post by zonotope »

sagichmal wrote: post #2

Not rolling your own consensus algorithm is approx. rules zero through ten of distributed systems programming. Nothing on this page provides any positive signal indicating unique (or even average) competence to the task. To wit, > We are borrowing all the best parts of existing technology and combining them in a unique and innovative way. To the best of my knowledge, none of the listed existing technologies have been…

We completely agree with you that, in general, you don't want to attempt to create an entirely new consensus mechanism. As you know, there are generally only 2 (perhaps 3) broad categories of byzantine tolerant consensus that boil down to PBFT and Nakamoto consensus. Ours is a flavor of PBFT designed to reduce message complexity. We also draw from Casper in order to get the incentives and game theory right which would not be considered part of the consensus algorithm traditionally, but must be considered in the DLT space. We're not really reinventing the wheel but rather adapting some interesting research to new problem set (which is consensus, without total ordering). We also plan on fully vetting and formally verifying our implementation before we move into production.

Re: Why we rolled our own consensus algorithm

#18

Post by eloff »

zzzcpan wrote: post #12
eloff wrote: post #7

Earlier quoted context omitted.

This seems like a really, really bad idea. However, if everyone had that attitude we'd never improve on the existing art. So I'm glad some people are willing to risk it. But just to reiterate, if you're thinking of rolling your own consensus - or even implementing an existing consensus algorithm yourself (although that's much worse for Paxos than for Raft) - you're almost certainly making a big mistake.

If you have enough background in distributed systems to choose a consensus algorithm you are definitely capable of implementing one yourself and even invent your own. It's not a bad idea either, most existing consensus-based systems are absolute garbage. People just hugely overestimate how much those algorithms matter for production systems. EDIT: Note, I'm not talking about blockchains and byzantine fault tolerance,…

That's like saying if you have some crypto background you can implement your own crypto library. It's TERRIBLE advice.

Read the Google paper on their chubby distributed locking system using paxos. They had a hard time getting it right - unless you think you're so much better than them. IIRC Leslie Lamport works (worked?) for them.

Use an existing, well used, well scrutinized implementation of a known algorithm.

Re: Why we rolled our own consensus algorithm

#19

Post by davidgerard »

cdiddy2 wrote: post #10
sagichmal wrote: post #2

Not rolling your own consensus algorithm is approx. rules zero through ten of distributed systems programming. Nothing on this page provides any positive signal indicating unique (or even average) competence to the task. To wit, > We are borrowing all the best parts of existing technology and combining them in a unique and innovative way. To the best of my knowledge, none of the listed existing technologies have been…

I believe bitcoins proof of work has been formally verified, however I can't find a source for that. There is also this https://iohk.io/research/papers/#9BKRHCSI that claims to be formally verified Proof of Stake.

Unless you're actually a computer scientist following the area ... all claims of solving the "blockchain trilemma" - decentralisation, security and efficiency - should be ignored unless and until they have an implementation to show you, and it survives the hostile Internet.

Promises are cheap.

e.g. IOHK is behind Cardano/ADA, which has a marvellous white paper making all sorts of mathematical promises ... and still runs off a central node.

In this case, it appears to be a system for private blockchains, i.e. a system with central administration. All "consensus" algorithms in this case are variants on taking turns - further elaboration is security theatre, given that attacking the system to that extent would go well beyond the legal agreements to be allowed to participate in the first place.

Re: Why we rolled our own consensus algorithm

#20

Post by davidgerard »

zonotope wrote: post #17
sagichmal wrote: post #2

Not rolling your own consensus algorithm is approx. rules zero through ten of distributed systems programming. Nothing on this page provides any positive signal indicating unique (or even average) competence to the task. To wit, > We are borrowing all the best parts of existing technology and combining them in a unique and innovative way. To the best of my knowledge, none of the listed existing technologies have been…

We completely agree with you that, in general, you don't want to attempt to create an entirely new consensus mechanism. As you know, there are generally only 2 (perhaps 3) broad categories of byzantine tolerant consensus that boil down to PBFT and Nakamoto consensus. Ours is a flavor of PBFT designed to reduce message complexity. We also draw from Casper in order to get the incentives and game theory right which woul…

So what's your system's threat model?

What will this achieve that a centralised database and existing legal structures and agreements won't?

(Assume I know the area and you can go into depth. However, remarkable claims will need a production system - anyone can and does claim anything's coming in "six months, for sure" - and an explanation of how you've achieved something that your many, many competitors haven't.)

Post reply on HN