Is this a Kafka competitor?
Facebook open-sources LogDevice, a distributed storage for sequential data
31–40 of 123 posts
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#32Very interesting!
I like the idea of decoupling compute from storage for streaming/log data.
I wonder if it would be easy to make it run under Consul, instead of ZooKeeper.
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#33Martin Kleppmann seems to point out technologies for problems
of similar patterns already exist - https://twitter.com/martinkl/status/1039938408393662465
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#34Earlier quoted context omitted.
How does LogDevice differ from Kafka?
Kafka brokers handle both the computation (partition/topic management, sequencing, assignments, etc) and storage together. This coupling creates scaling and operational challenges which LogDevice removes by separating the layers. Storage nodes can be as simple as object stores (but optimized for appending files) and use multiple non-deterministic locations for a given piece of data to randomize placement. They read,…
Interesting. Microsoft's Tango paper had some interesting things to say about sequencers/sequences as well.
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#35Martin Kleppmann seems to point out technologies for problems of similar patterns already exist - https://twitter.com/martinkl/status/1039938408393662465
Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference.
Do people just point their system journal at Kafka and wait for something to break?
At my previous job we built something similar to this out of rabbitmq and mongodb. I always wondered what the other big log companies used. Mongodb seemed like a pretty good fit, but a pure append only database might be even better. Trimming performance in MongoDB was subpar so we worked around it by creating a new collection for each day, trimming became a simple operation of dropping a collection at the end of each day.
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#36Martin Kleppmann seems to point out technologies for problems of similar patterns already exist - https://twitter.com/martinkl/status/1039938408393662465
Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference. Do people just point their system journal at Kafka and wait for something to break? At my previous job we built something similar to this out of rabbitmq and mongodb. I always wondered what the other big log companies used. Mongodb seemed like a pretty good fit, but a pure append only database might…
All of those are similar systems and have persistence. I'm not sure what distinction "streaming" makes but they also all support multiple publishers and subscribers. Some only use local storage on the nodes while others can tier out to cold storage like S3.
MongoDB is a full OLTP document store so it won't match the write throughput and pubsub features of these focused systems. RabbitMQ on the other hand has performance limits but is meant for complex service-bus style routing and RPC uses, but I recommend using NATS for that now.
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#37Thank to Open Source that, it looks a great project.
Could a LogDevice give a bit of informations about the scale they use that at facebook ?
- How many record this thing can injest per day ? - Any limitations on the maximum number of storage nodes ? - What would be your maximum and advise size of record for a production usage ? - ZooKeeper seems to be the center point used as epoch provider. Did you encounter any scaling limitations or max number of client due to that ?
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#38Happy to finally see LogDevice open. We have been working on this for years now.
Can you give an overview over the difference to eg Apache Kafka?
It seems very similar.
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#39Earlier quoted context omitted.
Scribe is the Facebook-internal Kafka equivalent. LogDevice is the storage layer used by Scribe. Scribe isn’t the only place where LogDevice is used though — Facebook has documented using it for TAO as well (as part of the secondary indices)
I don't believe Scribe and Kafka are equivalent. Isn't Scribe at-most-once? Unless we're talking about two different projects named Scribe, which is certainly possible.
Scribe doesn't really make decisions about that, it doesn't store checkpoints for readers. Readers are commonly more-than-once.
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#40The amount of great quality open source projects dein Facebook just keeps growing. I really like the consistency guarantees:
https://logdevice.io/docs/Concepts.html#consistency-guarante...
And it uses RocksDB under the hood:
https://logdevice.io/docs/Concepts.html#logsdb-the-local-log...