100% Java. I'm wondering. Why doesn't this has any Scala?
Because Scala adds overhead.
Twitter open-sources a high-performance replicated log service
21–30 of 121 posts
Re: Twitter open-sources a high-performance replicated log service
#22Earlier quoted context omitted.
Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "…
Seriously this seems like a common pattern in open source: [big company] could just improve [X] but instead builds something from the ground up.
Sometimes building something from the ground up allows you to make fundamental design decisions not available in the alternatives.
Re: Twitter open-sources a high-performance replicated log service
#23This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing di…
If you read the CS papers surrounding distributed systems, you will often see the notion of a 'journal' or a 'log', meaning an append-only structure, which typically contains numerous agreed-upon facts.
I agree that term describes the foundational data structure these services provide, but in common parlance that is not really what it means at all and it is confounded by a common(and boring) use case being to synchronize processing of actual, text-based, logs.
Re: Twitter open-sources a high-performance replicated log service
#24Earlier quoted context omitted.
Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "…
Seriously this seems like a common pattern in open source: [big company] could just improve [X] but instead builds something from the ground up.
Sometimes you need to let talented engineers build things from the ground up, because it's good for them, makes them happy, and stops them from going to work somewhere else. Keeping people with the skills to solve these types of problems around and happy is also great for recruiting and for helping your less capable engineers learn and grow.
Re: Twitter open-sources a high-performance replicated log service
#25Earlier quoted context omitted.
Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "…
Arguably there are benefits to developing in-house expertise, and no better way to develop expertise than to architect and build a solution end-to-end. Twitter now has several domain experts on staff who can continue maintaining DistributedLog and/or weigh its benefits against Kafka's and make more informed decisions going forward. Not saying that they couldn't have worked more closely with Kafka's team in the first…
> Arguably there are benefits to developing in-house expertise, and no better way to develop expertise than to architect and build a solution end-to-end
There are also drawbacks to consider: Those experts might just decide to leave and then you have an in-house solution that you have basically no chance to find experts on. At least with an open source solution you may have an easier time.
Re: Twitter open-sources a high-performance replicated log service
#26This looks potentially fantastic. If I could beg one wish from the developers of this (and almost every other project anywhere near this space), though, it would be one tiny piece of documentation:
What's your unique ID scheme?
Let's say I'm willing to believe[1] that you've got Durable and Consistent down, once messages make it committed in to the system. What's the story for messages on their way in? My application logs are buffered to the local disk, now I'm streaming them into central storage, and halfway through a TCP connection that's shuffled 2mb of thousands of messages into storage, the connection terminates -- unexpectedly, midmessage. Could the service have committed more messages than it acknowledged? Or many less than I've sent? Both could be true from the network standpoint.[2]
So, what I need to know, and what should be very easy to answer, front-and-center in your docs, pretty please:
1) Where should my log uploader resume?
2) Is there any danger of repeatedly entering some lines?
3) If I have log lines that are legitimately duplicates, will they be stored at the correct count?
These are questions that may have a different answer than the durability after data makes it fully into the system. It also may provide useful information about how complexity the code in a submitting client is, because good answers tend to require some kind of ID sequence being assigned on submitting clients, afaict. And it's really just plain critical to sanity.
----
[1] well, no, I'm not, "trust by verify" in all things etc etc; but let's suppose that's more believable and something I have to mechanically verify anyway, and doesn't have an obviously observable boolean at the protocol level as to whether it's going to work well or not, and system internals simply don't have such a sordid history of being over-simplified until they're broken like client interfaces so often are, so...! We'll handwave that to a later and more involved step of quality investigation.
Re: Twitter open-sources a high-performance replicated log service
#27By checking out the code quickly it looks like it's built on top of Apache Bookkeeper http://bookkeeper.apache.org/ ?
This blogpost https://blog.twitter.com/2015/building-distributedlog-twitte... mentioned somewhere else in the discussion confirms it's built on bookkeeper.
Re: Twitter open-sources a high-performance replicated log service
#28By checking out the code quickly it looks like it's built on top of Apache Bookkeeper http://bookkeeper.apache.org/ ?
More info on the stack: https://twitter.github.io/distributedlog/html/architecture/m...
Re: Twitter open-sources a high-performance replicated log service
#29Re: Twitter open-sources a high-performance replicated log service
#30Earlier quoted context omitted.
If you read the CS papers surrounding distributed systems, you will often see the notion of a 'journal' or a 'log', meaning an append-only structure, which typically contains numerous agreed-upon facts.
I agree that term describes the foundational data structure these services provide, but in common parlance that is not really what it means at all and it is confounded by a common(and boring) use case being to synchronize processing of actual, text-based, logs.
Jay Kreps, architect of Kafka, calls it a log.
https://engineering.linkedin.com/distributed-systems/log-wha...