I wish someone would explain why the licensing issue is not a pertinent discussion here And why those comments have been removed. In any other case that would be a perfectly reasonable point to consider.
LogDevice: a distributed data store for logs
31–40 of 42 posts
Re: LogDevice: a distributed data store for logs
#32Would someone from Facebook be able to comment on LogDevice versus (Twitter's) Apache Bookkeeper/DistributedLog?? The two sound very similar. I've been studying Bookkeeper this week to understand how it works and was excited to see the blogpost about LogDevice. My understanding isn't fully there yet, but BookKeeper seems to have a more involved protocol versus LogDevice: ie it has fencing to ensure that only one writ…
I'm on the LogDevice team at Facebook. Glad you're excited to hear about LogDevice!
I'm not familiar with Bookkeeper, so can't comment on that. I do know that, compared to most other systems, LogDevice emphasizes high write availability. So even if we're still sorting out the details of which records made it at the end of one epoch, we'll still take writes for future epochs. We just won't release them to readers until we've made enough copies of earlier records.
We do our best to ensure that only one sequencer is running at a time. We use Zookeeper to store information but the current epoch & sequencer, and whenever a new sequencer starts, it needs to talk to Zookeeper. So that helps with races where several machines want to become the sequencer for a single log at the same time.
Also, when clients are looking for the sequencer for a given log, they all try the same set of machines in the same order. Essentially there's a deterministic shuffling of the list of servers, seeded with the log id. So all clients will try to talk to the same server first, and only if that server is down, or learns that another sequencer is active, will the client try a different server.
Hope that helps!
Re: LogDevice: a distributed data store for logs
#33I wish someone would explain why the licensing issue is not a pertinent discussion here And why those comments have been removed. In any other case that would be a perfectly reasonable point to consider.
Because there is no licensing issue because this project hasn't been released. This blog post is a preannouncement. When it's released and its known what license it uses then it will become a pertinent discussion.
Re: LogDevice: a distributed data store for logs
#34Similarities to CORFU / Tango design (see Zlog post: http://noahdesu.github.io/2014/10/26/corfu-on-ceph.html).
Edit: This is a good thing
Re: LogDevice: a distributed data store for logs
#35A couple questions:
1. Do the log records pass through the sequencer nodes on the way to their destination, or does the writer issue a request for the sequence number and wait for a response?
2. If you're maintaining the current epoch number (and sequencer location) in Zookeeper, and a sequencer starts going up and down, you're definitely going to have problems, because nodes will get potentially stale epoch/location data from Zookeeper. Unless, that is, you have Zookeeper run a full consensus algorithm for every single query for every single record you write, which would kill performance.
So as long as you're using single sequencer nodes, and dealing with replication, and doing some kind of leader election to pick a new sequencer node when one goes down, why not just run a normal consensus algorithm (like Raft) for storing records? The Raft leader node serves as the sequencer, it takes care of replication, it tolerates failures, it will be no slower than what you're doing now, and it will be way more reliable.
Re: LogDevice: a distributed data store for logs
#36Reading the blog its not clear to me how they deal with gaps in the LSN sequence. The scalability & performance properties derive from 1) Using a separate sequencer that issues increasing sequence numbers, 2) Uncoordinated distributed writes of actual record value to storage nodes, 3) reconstitution of ordered log at consumer side. How does a consumer that have retrieved N and N+2 know if N+1 is not yet written, or i…
Hi, I work on the LogDevice team at Facebook. Within an epoch, the sequencer is a single process on a single machine and gives out LSNs sequentially. When a sequencer dies, and a new one is started, its first job is to fix up the end of the last epoch. If it can't find any copies of a given record, it inserts a "hole plug," to store the fact that the record is lost. So, except for hole plugs (which should be very rar…
Hi, thank you for the answer.
Can you comment on how writes to the LogDevice servers are performed - particularly if the writes have to go through the sequencer, or if the individual producers can write directly after obtaining a LSN?
Since a given LogDevice server will only receive a non-deterministic sub-sequence of records, I would think it has to receive its set of writes in order? (To have enough information to push to clients in order). Unless there is actually some mechanism by which it can determine the elements it should receive.
Trying to understand if your per-log throughput will be capped by the max traffic that can be pushed through a single sequencer, or the round-trip latency for waiting for acknowledged writes from LogDevice servers.
Re: LogDevice: a distributed data store for logs
#37I wish someone would explain why the licensing issue is not a pertinent discussion here And why those comments have been removed. In any other case that would be a perfectly reasonable point to consider.
Well, the underlying database RocksDB appears to have been relicensed under the Apache 2.0 license -- we'll have to see what Facebook chooses to do with this project when it's released. No sense getting the pitchforks out yet.
Re: LogDevice: a distributed data store for logs
#38Earlier quoted context omitted.
Hi, I work on the LogDevice team at Facebook. Within an epoch, the sequencer is a single process on a single machine and gives out LSNs sequentially. When a sequencer dies, and a new one is started, its first job is to fix up the end of the last epoch. If it can't find any copies of a given record, it inserts a "hole plug," to store the fact that the record is lost. So, except for hole plugs (which should be very rar…
Hi, thank you for the answer. Can you comment on how writes to the LogDevice servers are performed - particularly if the writes have to go through the sequencer, or if the individual producers can write directly after obtaining a LSN? Since a given LogDevice server will only receive a non-deterministic sub-sequence of records, I would think it has to receive its set of writes in order? (To have enough information to…
Trying to understand if your per-log throughput will be capped by the max traffic that can be pushed through a single sequencer,
Yes. You can think of an individual log like a database shard.
We have plans to allow the sequencer to just give out sequence numbers, and allow the clients to send the data directly to the storage nodes. But its not a high priority for us, since it's just a constant factor improvement. (Although a large one!) Users would still need to shard their data, although not as much.
Re: LogDevice: a distributed data store for logs
#39This approach to processing real-time and historical data works very well. We use a similar approach at Morgan Stanley and have a PL/compiler for real-time index construction and query processing on top (https://github.com/Morgan-Stanley/hobbes).
In addition to the general model of log writing + reactive processing, I've also found that we spend a lot of time focused on getting the _type structure_ of our data right (which allows for flexible/generic downstream processes that can also be efficient and type-checked against the structure of the data ahead of time). I wonder if/how this aspect of the problem has been considered by the Facebook team.
Re: LogDevice: a distributed data store for logs
#40At this stage, the only thing I care about is which license they are going to apply to this project. That'll be enough for me to know if I should bother investing any further time here.
Exactly. Specifically, whether it'll include the usual patent clause. I am sure it will and as such it will be of no interest to me.
Are there Facebook OSS projects without the patent clause?