Nice work! A Curator-like (http://curator.apache.org/) framework on top of LogCabin will make it immensely more useful for everyday production systems though.
Creator of Raft Algorithm introduces LogCabin
11–20 of 53 posts
Re: Creator of Raft Algorithm introduces LogCabin
#12First thoughts in my head.... 1) Ah, cool, creator of Raft algo, so some of the 'obvious' mistakes in an implementation should've been resolved by now (though if ppl weren't trying to use it in production.... who knows). 2) Great, C++, it should be efficient and fast with consistent RAM usage (Go's GC is a bit.... eh... still). 3) Oh, you need a C++ client library. :( I would love to say that API's don't matter, but…
I think I'd have to agree with you now: REST APIs seem to help with adoption. LogCabin was initially created for use with RAMCloud ( http://ramcloud.stanford.edu ), which mostly hand-rolls its RPC serialization to achieve its extreme performance goals (it budgets about 1 microsecond in software overhead per RPC). I thought I was being user-friendly in LogCabin by using protobufs, and at the time, something as embarrassingly slow as HTTP+JSON was unthinkable. I don't think it's too late to add a REST API to LogCabin, and it'd be pretty easy to make a REST proxy. Maybe that's worth doing for easier adoption from other languages. I also think the CLI client makes the barrier to entry pretty low for people that just want to test it out.
Re: Creator of Raft Algorithm introduces LogCabin
#13I'm not a C++ programmer, and work mainly with Go. I'm curious to know if it is usual for C++ developers to implement their own event loops for network transports, as Diego has done here [0]. The other example I know is Replicant [1], which is used by HyperDex, and it uses a custom event loop too [2]. [0] https://github.com/logcabin/logcabin/tree/master/Event [1] https://github.com/rescrv/Replicant [2] https://github…
This is fairly common. I've done professional work with Go and C++ and this is one of the biggest reasons I like Go over C++. C++'s concurrency primitives aren't quite good enough so you always need to build something on top of them. Using a library in C++ often requires grokking how it does concurrency and reconciling that with the way your code does concurrency. In Go everyone uses goroutines so you can very quickly understand how to use libraries.
Re: Creator of Raft Algorithm introduces LogCabin
#14I always wonder about cute names like this... is LogCabin so named because it's HouseBoat minus C? Or is it instead the integral of 1/Cabin dCabin?
Almost certainly has something to do with https://en.wikipedia.org/wiki/Timber_rafting :)
I've certainly been to that page before :)
I come from an academic lineage of log-based projects, from log-structured filesystems [1] which structure disks as a log, to RAMCloud [2][3][4] whose durability/recovery aspects are a distributed and partially in-memory extension of that, to Raft and LogCabin that are built around the concept of a replicated log for consensus.
LogCabin used to export a log-oriented data model, by the way, where the name made a bit more sense even. There was some talk of renaming it to TreeHouse now that it exports a key-value tree, but that one didn't really catch on.
[1] https://web.stanford.edu/~ouster/cgi-bin/papers/lfs.pdf
[2] http://ramcloud.stanford.edu
[3] https://www.usenix.org/conference/fast14/technical-sessions/...
[4] https://web.stanford.edu/~ouster/cgi-bin/papers/RumblePhd.pd...
Re: Creator of Raft Algorithm introduces LogCabin
#15What is the meaning of such system ? Why do people need something like zookeeper or LogCabin ? How does a coordinator came to play ? I don't know much about distribute system, but I would love to learn more...
You end up needing consensus for a lot of fault-tolerant systems that need to provide consistent results. For example, if your system allows users to choose their own usernames, and you're trying to guarantee that all usernames are unique, and your system needs to automatically deal with server failures, then I think you also need consensus.
Another way to think about it is that consensus gets you the equivalent of a compare-and-swap operation in a distributed setting. Just as compare-and-swap is useful for building synchronization primitives with shared memory, consensus is useful for building synchronization primitives across a network.
Re: Creator of Raft Algorithm introduces LogCabin
#16What is the meaning of such system ? Why do people need something like zookeeper or LogCabin ? How does a coordinator came to play ? I don't know much about distribute system, but I would love to learn more...
Consensus can be used for a variety of problems including membership, ordering, and atomic commits.
Re: Creator of Raft Algorithm introduces LogCabin
#17Congrats, how does compaction work? I assume you can serve kvs from memory?
Re: Creator of Raft Algorithm introduces LogCabin
#18Congrats, how does compaction work? I assume you can serve kvs from memory?
Oh and can I call this from c easily?
Re: Creator of Raft Algorithm introduces LogCabin
#19What is the meaning of such system ? Why do people need something like zookeeper or LogCabin ? How does a coordinator came to play ? I don't know much about distribute system, but I would love to learn more...
When you are building distributed systems often you need a way to coordinate between nodes. You can use a single node to do it, but then you have a single point of failure.
In case of Zookeeper you can utilize various recipes, like here: http://curator.apache.org/curator-recipes/index.html
You can even build more advanced systems on top of it:
https://bookkeeper.apache.org/index.html
https://cwiki.apache.org/confluence/display/BOOKKEEPER/HedWi...