Twitter open-sources a high-performance replicated log service
1–10 of 121 posts
Re: Twitter open-sources a high-performance replicated log service
#2How does this compare to Kafka?
Re: Twitter open-sources a high-performance replicated log service
#3This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware.
On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing discrete events in a reliable manner.
Re: Twitter open-sources a high-performance replicated log service
#4Anyone know how this similar and different from Kafka? Or why Twittered decide to build their own instead of using and contributing to Kafka?
Re: Twitter open-sources a high-performance replicated log service
#5Anyone know how this similar and different from Kafka? Or why Twittered decide to build their own instead of using and contributing to Kafka?
There's a great blog post on this: https://blog.twitter.com/2015/building-distributedlog-twitte...
Re: Twitter open-sources a high-performance replicated log service
#6Wonder how it compares to Facebook Scribe?
Re: Twitter open-sources a high-performance replicated log service
#7How does this compare to Kafka?
Here is a pdf going briefly into kafka comparison and design motivation http://goo.gl/J9XdsG . During my time at twitter I remember when Twitter switch from Kafka to distributedlog. They have an internal layer that adapts Kafka API to distributedlog, I am not sure if they have open sourced that
Re: Twitter open-sources a high-performance replicated log service
#8This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing di…
I don't think "distributed event processing framework" describes this system at all. It's exactly what it says on the tin: a replicated log (i.e. event log) storage system that one can build other systems on top of. It's not processing events as far as I can see?
Re: Twitter open-sources a high-performance replicated log service
#9Anyone know how this similar and different from Kafka? Or why Twittered decide to build their own instead of using and contributing to Kafka?
There's a great blog post on this: https://blog.twitter.com/2015/building-distributedlog-twitte...
Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]"
Seems reasonable, right?
Except "[3] Kafka addressed these durability concerns in version 0.8"
So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "do you guys have a plan and timeline to fix it?"
That's .... not great.
Now, maybe there are other reasons, but they aren't elucidated in this blog post :P.
Even then, my general view would be "did you approach the community and discuss your concerns or just dismiss them out of hand as infeasible", mainly because my experience is that if you do this, you often find they have exactly the same set of concerns/goals, and just need more resources to make it happen.
The desire to build shiny objects is very large. Outside of the paper plans of engineering teams, these things rarely end up up more shiny than what already exists or will be built by the time you are done.
Re: Twitter open-sources a high-performance replicated log service
#10Earlier quoted context omitted.
There's a great blog post on this: https://blog.twitter.com/2015/building-distributedlog-twitte...
Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "…
Or even "we are ready to pay 3 people for a year to build that, if we don't and give you the money instead what can you do ?".
The software industry has a real problem contributing to open source, the stuff, you know, allowing them to make money in the first place.