Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creator of Raft Algorithm introduces LogCabin

sourcegraph.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: Creator of Raft Algorithm introduces LogCabin

#31

Post by ongardie »

mavelikara wrote: post #27

I have a slightly off-topic question for Diego and other people experienced with distributed systems. Why are consensus algorithms always developed as systems, not as libraries? Zookeeper, etcd and LogCabin all operate as a cluster of processes which other nodes connect to over a client library. I can imagine that the distributed-state-machine-replication-mechanism of Raft or ZAB being implemented as a library where…

This question came up at the CoreOS Fest earlier today too. I think everyone wants libraries as well as services, and I'd like that too for LogCabin one day. It's just a bit harder when you start thinking about the details. It's not impossible though. As one example, CockroachDB is using etcd's Raft implementation as a library.

Part of the issue is that we want to have libraries with small simple interfaces, and a consensus library is probably going to be on the large side, as it has to interface with the disk, the network, and the state machine, plus membership changes, log compaction, who's the leader, configuration settings, debug logging, etc.

Another issue is that, as a library, it has to be in the language you're using. And if that language is C++ or similar, it has to be compatible/convenient with whatever threading approach you're using.

Then there's performance/memory. Some Raft implementations are designed to keep relatively small amounts of data in memory (like LogCabin or etcd), and others are meant to store large amounts on disk (like HydraBase). Some are optimized more for performance, others for safety.

I think we'll get libraries eventually. Keep in mind Raft is still very young. Paxos is around 26 years old, Raft is only .5 to 3 years old (depending on when you start counting). I like to think that Raft lowered the cost of developing a consensus system/library significantly, but it still takes time to develop mature implementations. Right now we have a lot of implementations in early stages; I wonder if some of these efforts will consolidate over time into really flexible libraries.

Re: Creator of Raft Algorithm introduces LogCabin

#32

Post by reubenbond »

mavelikara wrote: post #27

I have a slightly off-topic question for Diego and other people experienced with distributed systems. Why are consensus algorithms always developed as systems, not as libraries? Zookeeper, etcd and LogCabin all operate as a cluster of processes which other nodes connect to over a client library. I can imagine that the distributed-state-machine-replication-mechanism of Raft or ZAB being implemented as a library where…

Service Fabric does this. You provide the stream of operations and Fabric replicates them using distributed consensus.

Databases can be hosted on the framework. DocumentDB is, for example. While at MS, I wrote a near-real-time metrics system using Fabric and worked on a tunable-consistency distributed cache built on Fabric.

Summary of Service Fabric here: http://daprlabs.com/blog/blog/2015/04/30/service-fabric-2/

Re: Creator of Raft Algorithm introduces LogCabin

#33

Post by ongardie »

azinman2 wrote: post #23

What the webpage doesn't do is explain why I want this over the competitors. That would be helpful.

In part that's because I don't see other Raft implementations as competitors. A major goal in Raft was to enable many implementations. It'd be a huge fail if everyone switched to LogCabin and abandoned all the other Raft implementations.

Really though, it's probably not a feature list that'll make people want to use LogCabin. In Scale Computing's case, it's that they have a C++ code base that they could integrate LogCabin well with, and they know they can work with and maintain the code if they need to.

Re: Creator of Raft Algorithm introduces LogCabin

#34

Post by ongardie »

halayli wrote: post #29

This is a very inefficient way to handle events. https://github.com/logcabin/logcabin/blob/master/Event/Loop.... an event can be read/write, and each fd:event_type should be able to map to a different Event::file. In its current form, if you receive 2000 events, you'll need to unnecessarily context switch 2000 times.

Maybe, but I don't think it matters much. The comment there explains the issue, and I vaguely remember my measurements showing that this wasn't a big deal.

I do think it's an interesting (dare I say) limitation of epoll, and something I'd think about if I was redesigning epoll.

Please file an issue on github if you have ideas on how to improve this without adding significant complexity.

Re: Creator of Raft Algorithm introduces LogCabin

#35

Post by ongardie »

justinmk wrote: post #24
ongardie wrote: post #10

Earlier quoted context omitted.

LogCabin uses its event loop for network operations but then hands requests off to threads to process. I started out with libevent2, but the problem is it doesn't deal with having multiple threads very well (error-prone and inefficient). It's also not as well-documented as the man pages for epoll, so I ended up using epoll directly instead. What was really lost in the libevent2 -> epoll conversion was platform indepe…

> What was really lost in the libevent2 -> epoll conversion was platform independence Did you consider libuv?

I guess the answer is no, I haven't really looked at it. Anyone have experience with it in a large-ish project outside of Node?

Re: Creator of Raft Algorithm introduces LogCabin

#37

Post by mpercy »

ongardie wrote: post #10
misframer wrote: post #5

I'm not a C++ programmer, and work mainly with Go. I'm curious to know if it is usual for C++ developers to implement their own event loops for network transports, as Diego has done here [0]. The other example I know is Replicant [1], which is used by HyperDex, and it uses a custom event loop too [2]. [0] https://github.com/logcabin/logcabin/tree/master/Event [1] https://github.com/rescrv/Replicant [2] https://github…

LogCabin uses its event loop for network operations but then hands requests off to threads to process. I started out with libevent2, but the problem is it doesn't deal with having multiple threads very well (error-prone and inefficient). It's also not as well-documented as the man pages for epoll, so I ended up using epoll directly instead. What was really lost in the libevent2 -> epoll conversion was platform indepe…

We're using libev (actually libev++ which is cross-platform, has a decent underlying C impl, and a C++ bridge API). It seems to work well and is fast and cross platform. It can be a little tricky to figure out what is going on with the C++ API sometimes due to how the author tacked it on using macros.

Re: Creator of Raft Algorithm introduces LogCabin

#38

Post by bdarnell »

mavelikara wrote: post #27

I have a slightly off-topic question for Diego and other people experienced with distributed systems. Why are consensus algorithms always developed as systems, not as libraries? Zookeeper, etcd and LogCabin all operate as a cluster of processes which other nodes connect to over a client library. I can imagine that the distributed-state-machine-replication-mechanism of Raft or ZAB being implemented as a library where…

In my experience, consensus algorithms are developed as algorithms first, (reference) systems second, and libraries third. The reasons are simply that the algorithm is the most important part, and good libraries are hard. Diego has rightly focused on the algorithm, which has reached maturity before this LogCabin release.

I work on CockroachDB, which uses etcd's Raft implementation, so we are an existence proof that consensus libraries are possible. However, the library we share was the second attempt by the authors (Xiang Li and Yicheng Qin) to create a Raft implementation. The first attempt to develop a "library form" of a consensus algorithm is likely to reach a dead end, but over time it is possible to develop a library-style implementation.

Re: Creator of Raft Algorithm introduces LogCabin

#39

Post by apaprocki »

ongardie wrote: post #35
justinmk wrote: post #24

Earlier quoted context omitted.

> What was really lost in the libevent2 -> epoll conversion was platform independence Did you consider libuv?

I guess the answer is no, I haven't really looked at it. Anyone have experience with it in a large-ish project outside of Node?

Yes -- it is very straightforward to use and "just works". (It is also used in a similar way as found in Node in the luvit project: https://github.com/luvit/luvit)

Re: Creator of Raft Algorithm introduces LogCabin

#40

Post by jbandela1 »

Just the other day, I was looking for a raft consensus implementation in C++ and was disappointed at not finding any that were production ready (as opposed to Java and Go). This fits the need perfectly. Also, RamCloud says it requires ZooKeeper. Do you have any idea of the timeframe of when RamCloud will use LogCabin instead.
Post reply on HN