Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creator of Raft Algorithm introduces LogCabin

sourcegraph.com

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Creator of Raft Algorithm introduces LogCabin

#41

Post by tveita »

mavelikara wrote: post #27

I have a slightly off-topic question for Diego and other people experienced with distributed systems. Why are consensus algorithms always developed as systems, not as libraries? Zookeeper, etcd and LogCabin all operate as a cluster of processes which other nodes connect to over a client library. I can imagine that the distributed-state-machine-replication-mechanism of Raft or ZAB being implemented as a library where…

Well etcd started out using the goraft library, so this isn't always true, but I can think of some reasons I'd prefer to have the quorum as a separate system:

Writing it as a service lets you implement it in once in a high-level language like Java or Go and use it from all languages. To get the same portability in a library you'd have to write it in C. It's hard enough to maintain consistent data without worrying about memory corruption bugs.

The protocol implementation is only a part of durable consensus, you also need durable storage, sensible network timeouts, shutdown handling, etc.

You usually want different configurations for your application and your quorum. For the quorum you usually want 3, 5 or 7 members, while your application could be anywhere from one to hundreds of instances. Your quorum members must always know how many other members are supposed to be in the quorum, while your application could remove or add instances on the fly. For the quorum you need low latency, while you may want to optimise your application for throughput. (e.g. disks, garbage collection, swapping)

Re: Creator of Raft Algorithm introduces LogCabin

#43

Post by dkhenry »

misframer wrote: post #5

I'm not a C++ programmer, and work mainly with Go. I'm curious to know if it is usual for C++ developers to implement their own event loops for network transports, as Diego has done here [0]. The other example I know is Replicant [1], which is used by HyperDex, and it uses a custom event loop too [2]. [0] https://github.com/logcabin/logcabin/tree/master/Event [1] https://github.com/rescrv/Replicant [2] https://github…

C++ developers don't make their own event loop. They use epoll which is provided by the kernel. That is still the area the C/C++ ( and Rust ) have over Go, they can use syscalls without requiring the goodwill of one of the language developers.

Re: Creator of Raft Algorithm introduces LogCabin

#44

Post by ongardie »

jbandela1 wrote: post #40

Just the other day, I was looking for a raft consensus implementation in C++ and was disappointed at not finding any that were production ready (as opposed to Java and Go). This fits the need perfectly. Also, RamCloud says it requires ZooKeeper. Do you have any idea of the timeframe of when RamCloud will use LogCabin instead.

Good timing then, and I'd be happy to talk more about whether LogCabin is a good fit for your use.

RAMCloud used to depend on an earlier version of LogCabin (before the data model was a key-value tree), then John rewrote the RAMCloud end of that and switched it to using ZooKeeper, but he made it pluggable on the RAMCloud side. So we just need an "ExternalStorage" implementation for LogCabin in RAMCloud to make that work again. It should be relatively straightforward and I'm confident they'd welcome a well-written and tested patch, but no one has volunteered yet.

Re: Creator of Raft Algorithm introduces LogCabin

#45

Post by Manishearth »

mavelikara wrote: post #27

I have a slightly off-topic question for Diego and other people experienced with distributed systems. Why are consensus algorithms always developed as systems, not as libraries? Zookeeper, etcd and LogCabin all operate as a cluster of processes which other nodes connect to over a client library. I can imagine that the distributed-state-machine-replication-mechanism of Raft or ZAB being implemented as a library where…

The Rust Raft library[0] (not my project) is a library; you must specify how the store[1]/statemachine[2] works under the hood, though it comes with default implementations for both. I'd like for it to abstract over communication channels too, but currently it's tightly wired to an RPC and IO library and I don't have the time to try and fix that.

[0]: https://hoverbear.github.io/raft/raft/index.html

[1]: https://hoverbear.github.io/raft/raft/store/trait.Store.html

[2]: https://hoverbear.github.io/raft/raft/state_machine/trait.St...

Re: Creator of Raft Algorithm introduces LogCabin

#46

Post by ongardie »

moomin wrote: post #36

Reminder that no system is production ready until aphyr has had time to break it. :)

Yes, good point. I welcome Mr. McCaffrey to give it a spin, if he's so willing. And though there may be critical bugs left to find, I don't feel like I've misrepresented the current state of LogCabin.

Re: Creator of Raft Algorithm introduces LogCabin

#48

Post by ongardie »

0xdeadbeefbabe wrote: post #47

In raft how can a client know it is receiving stale, yet agreed upon, data? I'm talking about the edge case of having a network partition or some other condition like it.

Raft implementations can choose to implement reads in various ways. In LogCabin reads are linearizable, meaning that the results are current as of sometime after the read request was initiated, and it doesn't rely on bounded clock drift to make this guarantee. That's about the best you can do in terms of freshness in a distributed system. Other implementations might offer weaker read semantics or give the client the option. Check out 6.3-6.4 in my dissertation for a ton more detail: https://github.com/ongardie/dissertation#readme

Re: Creator of Raft Algorithm introduces LogCabin

#49

Post by jerf »

mavelikara wrote: post #27

I have a slightly off-topic question for Diego and other people experienced with distributed systems. Why are consensus algorithms always developed as systems, not as libraries? Zookeeper, etcd and LogCabin all operate as a cluster of processes which other nodes connect to over a client library. I can imagine that the distributed-state-machine-replication-mechanism of Raft or ZAB being implemented as a library where…

A "library" that wants to have a process of some sort continuously running encounters the problem that the more cross-language generic you try to make the library, the less possible it is to work out a threading model that can be used by everyone, and the less language-generic you make it, the smaller your target audience is, which has non-linear affects on the usage and development resources it can attract. Languages that include a "blessed" runtime model like Node or Go can easily ship such "daemonized libraries", but then they are generally impossible to bind to for anybody else. Using POSIX threads, by contrast, will lock a lot of other languages out that have very sophisticated runtimes, be very difficult to use in others, and, even in C, when you get right down to it, there's no guarantee that they'll play nicely with the rest of the C program.

Distributing a service invokes the sort of lowest-common-denominator solution to this problem, which is "OS process". Everyone can talk to an OS process.

(This is an explanation, not advocacy or celebration.)

Re: Creator of Raft Algorithm introduces LogCabin

#50

Post by 0xdeadbeefbabe »

ongardie wrote: post #48
0xdeadbeefbabe wrote: post #47

In raft how can a client know it is receiving stale, yet agreed upon, data? I'm talking about the edge case of having a network partition or some other condition like it.

Raft implementations can choose to implement reads in various ways. In LogCabin reads are linearizable, meaning that the results are current as of sometime after the read request was initiated, and it doesn't rely on bounded clock drift to make this guarantee. That's about the best you can do in terms of freshness in a distributed system. Other implementations might offer weaker read semantics or give the client the…

I'm still misunderstanding what happens when a network partition creates a two leader scenario. It is in a slide here http://thesecretlivesofdata.com/raft/ under "log replication" after creating a network partition that separates CDE from AB. Doesn't that mean a client of node B could read stale data? Or maybe an implementation like etcd would let the client choose to read or not in that case?
Post reply on HN