Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter open-sources a high-performance replicated log service

github.com

Post reply on HN
Print view View original

11–20 of 121 posts

Re: Twitter open-sources a high-performance replicated log service

#11

Post by tcoppi »

allengeorge wrote: post #8
tcoppi wrote: post #3

This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing di…

I don't think "distributed event processing framework" describes this system at all. It's exactly what it says on the tin: a replicated log (i.e. event log) storage system that one can build other systems on top of. It's not processing events as far as I can see?

You put events in one end, and it distributes them, reliably and consistently, to distributed end-points that can consume them. That is far more powerful than simply being used to log events for later retrieval, though it can be used for that. I have seen systems using Kafka that process tens of thousands of time-sensitive events a second and distribute them all over the world for independent processing as part of a larger system.

Re: Twitter open-sources a high-performance replicated log service

#12

Post by d0vs »

DannyBee wrote: post #9
murph wrote: post #5

Earlier quoted context omitted.

There's a great blog post on this: https://blog.twitter.com/2015/building-distributedlog-twitte...

Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "…

Seriously this seems like a common pattern in open source: [big company] could just improve [X] but instead builds something from the ground up.

Re: Twitter open-sources a high-performance replicated log service

#15

Post by toomuchtodo »

tcoppi wrote: post #11
allengeorge wrote: post #8

Earlier quoted context omitted.

I don't think "distributed event processing framework" describes this system at all. It's exactly what it says on the tin: a replicated log (i.e. event log) storage system that one can build other systems on top of. It's not processing events as far as I can see?

You put events in one end, and it distributes them, reliably and consistently, to distributed end-points that can consume them. That is far more powerful than simply being used to log events for later retrieval, though it can be used for that. I have seen systems using Kafka that process tens of thousands of time-sensitive events a second and distribute them all over the world for independent processing as part of a…

"real-time event journaling and dissemination system"?

Re: Twitter open-sources a high-performance replicated log service

#17

Post by goejoegg »

tcoppi wrote: post #3

This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing di…

If you read the CS papers surrounding distributed systems, you will often see the notion of a 'journal' or a 'log', meaning an append-only structure, which typically contains numerous agreed-upon facts.

Re: Twitter open-sources a high-performance replicated log service

#18

Post by tcoppi »

toomuchtodo wrote: post #15
tcoppi wrote: post #11

Earlier quoted context omitted.

You put events in one end, and it distributes them, reliably and consistently, to distributed end-points that can consume them. That is far more powerful than simply being used to log events for later retrieval, though it can be used for that. I have seen systems using Kafka that process tens of thousands of time-sensitive events a second and distribute them all over the world for independent processing as part of a…

"real-time event journaling and dissemination system"?

Yes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.

Re: Twitter open-sources a high-performance replicated log service

#19

Post by [deleted] »

tcoppi wrote: post #18
toomuchtodo wrote: post #15

Earlier quoted context omitted.

"real-time event journaling and dissemination system"?

Yes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.

[deleted]

Re: Twitter open-sources a high-performance replicated log service

#20

Post by Smudge »

DannyBee wrote: post #9
murph wrote: post #5

Earlier quoted context omitted.

There's a great blog post on this: https://blog.twitter.com/2015/building-distributedlog-twitte...

Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "…

Arguably there are benefits to developing in-house expertise, and no better way to develop expertise than to architect and build a solution end-to-end. Twitter now has several domain experts on staff who can continue maintaining DistributedLog and/or weigh its benefits against Kafka's and make more informed decisions going forward.

Not saying that they couldn't have worked more closely with Kafka's team in the first place, but, hey, now we have two Kafkaesque log services instead of just one. Seems like a win to me.

Post reply on HN