21–30 of 123 posts
Earlier quoted context omitted.
Is there supposed to be a replay of that talk on the site you link to or is it just not loading for me?
The event is hosted by Facebook and there is an embedded Facebook video player on that page. Here's the direct link: https://www.facebook.com/atscaleevents/videos/19602876909109...
Great to see this released. Some similar architecture decisions to Apache Pulsar as well with the separate of compute (in this case the sequencer) from the storage. Kafka has done well so far, especially in making streaming systems more common, but it's about time for the next-gen systems.
How does LogDevice differ from Kafka?
Scribe isn’t the only place where LogDevice is used though — Facebook has documented using it for TAO as well (as part of the secondary indices)
Happy to finally see LogDevice open. We have been working on this for years now.
Great to see this released. Some similar architecture decisions to Apache Pulsar as well with the separate of compute (in this case the sequencer) from the storage. Kafka has done well so far, especially in making streaming systems more common, but it's about time for the next-gen systems.
How does LogDevice differ from Kafka?
I'm not that familiar with Kafka, but in general LogDevice emphasizes write availability over read availability. There are many applications where data is being generated all the time, and if you don't write it, it will be lost. However, if reading is delayed, it just means readers are a little behind and will need to catch up.
So, when a sequencer node dies and we need to figure out what happened to the records that were in flight -- which ones ended up on disk & can be replicated, what the last record was -- LogDevice still accepts new writes. However, to ensure ordering, these new writes aren't visible to readers until the earlier writes are sorted out.
Earlier quoted context omitted.
How does LogDevice differ from Kafka?
Scribe is the Facebook-internal Kafka equivalent. LogDevice is the storage layer used by Scribe. Scribe isn’t the only place where LogDevice is used though — Facebook has documented using it for TAO as well (as part of the secondary indices)
Unless we're talking about two different projects named Scribe, which is certainly possible.
What benefit to facebook is there from open sourcing technology they have developed?
So there's no cost to open sourcing. The benefit comes from being known as technically innovative in general, and for recruiting, being known as having interesting, meaty, challenging projects to work on.
The impetus usually comes from team members who want to do the work. It could be to become known for having worked on the project, or a sense of giving back to the community, or a hope that you'll get bug fixes & features from outside contributors. In my (very limited) experience, managers "passively encourage" it -- they generally don't push the team to do it, but when the team asks, they encourage it.
What benefit to facebook is there from open sourcing technology they have developed?
What benefit to facebook is there from open sourcing technology they have developed?
The cost of maintaining an open source project is real, but when it is a world-class piece of infrastructure, open sourcing it helps keep it world-class.
What benefit to facebook is there from open sourcing technology they have developed?
Facebook's competitive advantage doesn't come from having the best reliable streaming data store at scale, or from its software in general. Even if MySpace, Friendster or Google + got their hands on the whole software stack & started running it, people would stick with Facebook. So there's no cost to open sourcing. The benefit comes from being known as technically innovative in general, and for recruiting, being know…
Not true, there's a legal cost associated with making sure something is really ready for public eyes.