Nice to see they're storing my baby pictures and food choices forever.
Facebook open-sources LogDevice, a distributed storage for sequential data
11–20 of 123 posts
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#12Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#13Great to see this released. Some similar architecture decisions to Apache Pulsar as well with the separate of compute (in this case the sequencer) from the storage. Kafka has done well so far, especially in making streaming systems more common, but it's about time for the next-gen systems.
How does LogDevice differ from Kafka?
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#14How does that fit in a ML training pipeline?
(this is mentioned on the page)
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#15Great to see this released. Some similar architecture decisions to Apache Pulsar as well with the separate of compute (in this case the sequencer) from the storage. Kafka has done well so far, especially in making streaming systems more common, but it's about time for the next-gen systems.
How does LogDevice differ from Kafka?
I’m wondering this as well. In the description it says that it ensures total ordering while Kafka only ensures partition ordering. I haven’t read enough to say more.
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#16Great to see this released. Some similar architecture decisions to Apache Pulsar as well with the separate of compute (in this case the sequencer) from the storage. Kafka has done well so far, especially in making streaming systems more common, but it's about time for the next-gen systems.
How does LogDevice differ from Kafka?
Kafka brokers handle both the computation (partition/topic management, sequencing, assignments, etc) and storage together. This coupling creates scaling and operational challenges which LogDevice removes by separating the layers. Storage nodes can be as simple as object stores (but optimized for appending files) and use multiple non-deterministic locations for a given piece of data to randomize placement. They read, write and recover data very quickly by working together in a mesh.
Meanwhile the compute layer becomes very lightweight and almost stateless, which is easy to scale. In LogDevice, the Sequencers are potential bottlenecks but generating a series of incrementing numbers is about the fastest thing you can do so it'll outpace any actual data ingest to a single log, while giving you a total order of all entries within that log. The numbers (LSNs) follow the Hi/Lo sequence pattern so if a Sequencer fails, another one takes its place with a greater "High" number, so it's guaranteed that all of its LSNs will be greater than the previous Sequencer as a result. This also provides a built-in buffer to still accept messages and assign the permanent LSNs to them after recovery in case a Sequencer fails.
Apache Pulsar is similar to LogDevice but goes further where brokers manage connections, routing and message acknowledgements while data is sent to a separate layer of Apache Bookkeeper nodes which store the data in append-optimized log files.
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#17How does that fit in a ML training pipeline? (this is mentioned on the page)
It's just streaming data but more scalable and with total ordering which can be important for ML.
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#18Very interesting! I hadn't heard of this before but I'd love to see it in action.
If anyone from the FB team or anyone using LogDevice wants to test performance with Optane SSDs (and compare to a NAND SSD), make a request by submitting an issue on our GitHub page: https://github.com/AccelerateWithOptane/lab/issues. I'll hook you up with a server hosted by Packet.
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#19Awesome, I have been waiting for this since seeing the @scale talk about it. https://atscaleconference.com/videos/logdevice-a-file-struct...
Is there supposed to be a replay of that talk on the site you link to or is it just not loading for me?
Re: Facebook open-sources LogDevice, a distributed storage for sequential data
#20Awesome, I have been waiting for this since seeing the @scale talk about it. https://atscaleconference.com/videos/logdevice-a-file-struct...
Is there supposed to be a replay of that talk on the site you link to or is it just not loading for me?
The event is hosted by Facebook and there is an embedded Facebook video player on that page. Here's the direct link: https://www.facebook.com/atscaleevents/videos/19602876909109...