Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter open-sources a high-performance replicated log service

github.com

Post reply on HN
Print view View original

31–40 of 121 posts

Re: Twitter open-sources a high-performance replicated log service

#31

Post by SEJeff »

tcoppi wrote: post #3

This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing di…

I'm pretty sure that this post inspired DL. It was written by Jay, one of the 3 founders of kafka (Jay, Jun, and Neha) and should be recommended reading for every software engineer if you've not read it:

https://engineering.linkedin.com/distributed-systems/log-wha...

An ordered append only datastructure is rightfully called a log. The fact that text based files are called logs is just an annoying feature in common english usage.

Re: Twitter open-sources a high-performance replicated log service

#33

Post by gshx »

scaleout1 wrote: post #7
tlrobinson wrote: post #2

How does this compare to Kafka?

Here is a pdf going briefly into kafka comparison and design motivation http://goo.gl/J9XdsG . During my time at twitter I remember when Twitter switch from Kafka to distributedlog. They have an internal layer that adapts Kafka API to distributedlog, I am not sure if they have open sourced that

Very nice, now that we have some good competition in this space - kafka, ddl and wormhole. If only Fb can open src wormhole, there will be some good related systems software to choose from and contribute to.

Re: Twitter open-sources a high-performance replicated log service

#34

Post by Terr_ »

tcoppi wrote: post #18
toomuchtodo wrote: post #15

Earlier quoted context omitted.

"real-time event journaling and dissemination system"?

Yes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.

Somehow I associate "journal" with temporary/transitional data, but that's probably just me thinking of journaling file-systems.

Re: Twitter open-sources a high-performance replicated log service

#35

Post by niccaluim »

merb wrote: post #13

100% Java. I'm wondering. Why doesn't this has any Scala?

Rumors of Scala's dominance at Twitter are slightly exaggerated. While it's true that the non-revenue-related backend is almost all Scala, on the ads eng side, it's almost 100% Java. Which language a new project is written in has more to do with who's writing it than anything else.

Re: Twitter open-sources a high-performance replicated log service

#38

Post by toomuchtodo »

Terr_ wrote: post #34
tcoppi wrote: post #18

Earlier quoted context omitted.

Yes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.

Somehow I associate "journal" with temporary/transitional data, but that's probably just me thinking of journaling file-systems.

When I think journal, I think a permanent record of rdbms transactions or perhaps even a blockchain. I agree with you though that the persistence of data needs to be defined when discussing a journal of any sorts.
Post reply on HN